Jul 22 16:18:55.436319 ip-10-0-133-136 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 16:18:55.436332 ip-10-0-133-136 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 16:18:55.436339 ip-10-0-133-136 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 16:18:55.436558 ip-10-0-133-136 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 16:19:05.685849 ip-10-0-133-136 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 16:19:05.685870 ip-10-0-133-136 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ab28dcd74d28464c850e2035588843db -- Jul 22 16:20:41.150793 ip-10-0-133-136 systemd[1]: Starting Kubernetes Kubelet... Jul 22 16:20:41.598904 ip-10-0-133-136 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:20:41.598904 ip-10-0-133-136 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 16:20:41.598904 ip-10-0-133-136 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:20:41.598904 ip-10-0-133-136 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 16:20:41.598904 ip-10-0-133-136 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:20:41.598904 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.595384 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603030 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603042 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603045 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603049 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603052 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603055 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603058 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603061 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603063 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603066 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603068 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603071 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603077 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603081 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603083 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603086 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603089 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603091 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603095 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:20:41.615058 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603098 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603100 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603103 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603108 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603111 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603115 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603118 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603121 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603124 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603128 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603131 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603134 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603136 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603139 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603142 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603145 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603148 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603151 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:20:41.629686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603153 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603156 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603159 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603161 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603164 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603167 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603169 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603172 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603175 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603178 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603180 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603183 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603185 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603188 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603198 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603200 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603203 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603205 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603208 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603211 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:20:41.631509 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603213 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603216 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603218 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603221 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603223 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603226 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603229 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603231 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603233 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603238 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603241 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603243 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603246 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603249 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603254 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603257 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603259 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603262 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603265 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603267 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:20:41.633898 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603270 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603272 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603275 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603278 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603280 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603283 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603285 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603288 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603290 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603294 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603297 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603299 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603303 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603307 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603310 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603313 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603316 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603319 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603321 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:20:41.649307 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603324 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603327 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.603329 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604050 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604057 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604060 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604063 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604065 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604068 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604071 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604074 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604076 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604079 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604083 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604085 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604088 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604090 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604093 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604095 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604098 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:20:41.651581 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604101 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604104 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604106 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604109 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604111 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604114 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604116 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604119 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604121 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604124 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604126 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604129 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604131 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604134 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604137 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604140 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604142 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604145 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604149 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:20:41.653512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604153 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604157 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604161 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604164 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604167 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604170 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604173 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604176 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604179 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604183 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604186 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604189 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604192 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604195 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604198 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604200 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604203 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604206 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604208 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604211 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:20:41.655863 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604213 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604216 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604221 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604224 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604226 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604229 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604232 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604234 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604237 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604239 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604242 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604245 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604248 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604250 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604252 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604256 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604259 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604262 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604264 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604267 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:20:41.669288 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604270 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604273 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604275 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604278 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604280 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604283 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604285 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604288 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604291 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604294 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604296 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604299 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604301 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604304 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604309 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604313 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604316 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604318 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604321 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:20:41.671567 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604324 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604327 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604330 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604332 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604696 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604701 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604704 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604707 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604711 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604715 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604718 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604721 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604724 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604726 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604729 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604732 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604734 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604737 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604739 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604742 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:20:41.673430 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604745 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604747 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604750 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604752 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604755 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604758 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604761 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604765 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604768 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604772 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604776 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604779 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604782 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604786 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604789 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604791 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604794 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604797 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604799 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604802 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:20:41.675920 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604805 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604808 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604811 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604813 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604816 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604818 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604823 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604825 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604828 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604830 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604833 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604835 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604838 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604840 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604843 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604845 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604848 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604850 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604853 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604857 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:20:41.748928 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604860 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604862 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604865 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604867 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604870 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604873 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604877 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604879 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604882 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604884 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604887 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604889 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604892 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604894 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604901 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604904 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604906 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604925 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604930 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604933 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:20:41.776121 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604935 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:20:41.686258 ip-10-0-133-136 systemd[1]: Started Kubernetes Kubelet. Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604938 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604941 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604944 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604946 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604949 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604952 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604955 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604957 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604960 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604963 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604966 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604969 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604972 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604975 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604978 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604981 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604984 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604987 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:20:41.980353 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604990 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604993 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604995 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.604998 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605062 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605069 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605076 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605080 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605084 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605088 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605091 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605104 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605107 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605110 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605114 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605118 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605121 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605124 2580 flags.go:64] FLAG: --cgroup-root="" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605126 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605129 2580 flags.go:64] FLAG: --client-ca-file="" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605132 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605135 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605139 2580 flags.go:64] FLAG: --cluster-domain="" Jul 22 16:20:41.982409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605142 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605146 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605149 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605152 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605156 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605159 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605162 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605165 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605168 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605171 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605174 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605177 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605180 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605184 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605187 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605190 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605192 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605196 2580 flags.go:64] FLAG: --enable-server="true" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605199 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605204 2580 flags.go:64] FLAG: --event-burst="100" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605208 2580 flags.go:64] FLAG: --event-qps="50" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605211 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605214 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605217 2580 flags.go:64] FLAG: --eviction-hard="" Jul 22 16:20:41.984748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605220 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605223 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605226 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605229 2580 flags.go:64] FLAG: --eviction-soft="" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605232 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605234 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605237 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605240 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605243 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605247 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605250 2580 flags.go:64] FLAG: --feature-gates="" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605253 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605257 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605260 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605263 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605266 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605269 2580 flags.go:64] FLAG: --help="false" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605271 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-133-136.ec2.internal" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605274 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605277 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605280 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605284 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605287 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605290 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 16:20:41.986803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605293 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605295 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605298 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605302 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605306 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605309 2580 flags.go:64] FLAG: --kube-reserved="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605312 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605314 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605317 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605320 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605323 2580 flags.go:64] FLAG: --lock-file="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605325 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605328 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605331 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605336 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605338 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605341 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605346 2580 flags.go:64] FLAG: --logging-format="text" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605349 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605352 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605355 2580 flags.go:64] FLAG: --manifest-url="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605357 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605361 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605365 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605369 2580 flags.go:64] FLAG: --max-pods="110" Jul 22 16:20:41.987796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605371 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605375 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605378 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605381 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605384 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605386 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605389 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605396 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605399 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605402 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605405 2580 flags.go:64] FLAG: --pod-cidr="" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605410 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605416 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605418 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605421 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605424 2580 flags.go:64] FLAG: --port="10250" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605427 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605430 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c5d26643e69d72fa" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605433 2580 flags.go:64] FLAG: --qos-reserved="" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605436 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605439 2580 flags.go:64] FLAG: --register-node="true" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605442 2580 flags.go:64] FLAG: --register-with-taints="" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605445 2580 flags.go:64] FLAG: --registry-burst="10" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605449 2580 flags.go:64] FLAG: --registry-qps="5" Jul 22 16:20:41.989865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605451 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605455 2580 flags.go:64] FLAG: --reserved-memory="" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605459 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605462 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605465 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605467 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605470 2580 flags.go:64] FLAG: --runonce="false" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605473 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605476 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605479 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605482 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605485 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605487 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605504 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605507 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605510 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605513 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605515 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605518 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605522 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605526 2580 flags.go:64] FLAG: --system-cgroups="" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605529 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605534 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605537 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605540 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 16:20:41.990995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605544 2580 flags.go:64] FLAG: --tls-min-version="" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605546 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605549 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605552 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605555 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605558 2580 flags.go:64] FLAG: --v="2" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605561 2580 flags.go:64] FLAG: --version="false" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605565 2580 flags.go:64] FLAG: --vmodule="" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605569 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.605573 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605704 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605710 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605713 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605717 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605719 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605722 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605725 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605727 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605730 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605732 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605735 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605738 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:20:41.992308 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605740 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605743 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605745 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605748 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605752 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605755 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605757 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605761 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605763 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605766 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605768 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605771 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605773 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605776 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605779 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605781 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605784 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605786 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605790 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:20:41.993266 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605792 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605795 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605798 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605801 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605803 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605806 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605809 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605811 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605815 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605818 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605821 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605824 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605826 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605829 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605832 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605834 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605837 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605842 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605844 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605847 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:20:41.994060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605850 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605852 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605855 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605857 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605860 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605862 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605865 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605867 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605870 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605872 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605875 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605881 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605884 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605887 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605891 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605894 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605908 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605911 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605914 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:20:41.994842 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605916 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605919 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605922 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605925 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605927 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605930 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605932 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605935 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605938 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605940 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605945 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605948 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605950 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605953 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605956 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605958 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605961 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605964 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605966 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605969 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:20:41.995420 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605971 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605974 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605977 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605979 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605983 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605985 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605988 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605990 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.605993 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.606919 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.615102 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.615116 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.616001 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.616008 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616073 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:20:41.996018 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616079 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616082 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616085 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616088 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616090 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616093 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616096 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616102 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616105 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616108 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616111 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616114 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616117 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616120 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616122 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616125 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616127 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616130 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616133 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616135 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:20:41.996553 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616137 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616140 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616144 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616147 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616149 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616152 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616154 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616157 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616160 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616163 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616166 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616170 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616174 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616176 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616179 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616182 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616184 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616187 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616189 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616193 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:20:41.997103 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616196 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616198 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616203 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616206 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616209 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616212 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616215 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616217 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616220 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616223 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616225 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616228 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616231 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616233 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616237 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616239 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616242 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616245 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616248 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616250 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:20:41.997824 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616253 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616256 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616260 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616263 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616266 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616269 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616272 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616274 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616277 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616280 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616282 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616286 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616289 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616292 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616294 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616297 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616300 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616302 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:20:41.998366 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616305 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616307 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616310 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616313 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616315 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616318 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616320 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616323 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616327 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616330 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616332 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616335 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616337 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616340 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616343 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616345 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616348 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616350 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616353 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:20:41.998916 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616355 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.616360 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616470 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616475 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616478 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616480 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616484 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616487 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616504 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616509 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616513 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616516 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616519 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616522 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616525 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:20:41.999448 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616528 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616530 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616533 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616536 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616538 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616541 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616543 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616547 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616550 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616553 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616556 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616558 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616560 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616563 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616565 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616568 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616570 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616573 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616575 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616578 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616581 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:20:41.999979 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616583 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616586 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616590 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616592 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616595 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616597 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616599 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616602 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616604 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616607 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616610 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616612 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616615 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616617 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616621 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616624 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616628 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616631 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616636 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:20:42.000875 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616639 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616643 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616646 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616648 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616651 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616654 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616656 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616659 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616662 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616664 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616667 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616669 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616672 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616674 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616677 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616683 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616685 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616688 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616691 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616693 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:20:42.001441 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616696 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616698 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616701 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616703 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616706 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616708 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616711 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616713 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616716 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616719 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616721 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616726 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616729 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616731 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616733 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616736 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616739 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616741 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616744 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:20:42.200087 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616746 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616749 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616751 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616754 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616757 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616759 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:41.616762 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.616766 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.616862 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.619244 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.620970 2580 server.go:1019] "Starting client certificate rotation" Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.621068 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.622052 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:20:42.200734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.624019 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:20:42.201101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.631115 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:20:42.201101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.645295 2580 log.go:25] "Validated CRI v1 runtime API" Jul 22 16:20:42.201101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.647767 2580 log.go:25] "Validated CRI v1 image API" Jul 22 16:20:42.201101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.649720 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 16:20:42.201101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.652305 2580 fs.go:135] Filesystem UUIDs: map[2c334ba6-b4f5-4de9-819c-52239767bdeb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3583d6d-2e21-4492-b426-90bf49d5d79d:/dev/nvme0n1p3] Jul 22 16:20:42.201101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.652324 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.658592 2580 manager.go:216] Machine: {Timestamp:2026-07-22 16:20:41.65719733 +0000 UTC m=+0.375193407 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099866 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c44a25e3b9ff0db2df35d06d07b44 SystemUUID:ec2c44a2-5e3b-9ff0-db2d-f35d06d07b44 BootID:ab28dcd7-4d28-464c-850e-2035588843db Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:3e:66:f2:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:3e:66:f2:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:85:e1:2c:52:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.658694 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.658776 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.662195 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.662218 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-136.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.662457 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.663161 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.663173 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.663181 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.663194 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.665121 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.669940 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670058 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670086 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670095 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670115 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 16:20:42.201377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670126 2580 kubelet.go:398] "Adding apiserver pod source" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670139 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670537 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.670556 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.673789 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.677624 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.677651 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679179 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679193 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679200 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679205 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679214 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679221 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679233 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679240 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679253 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679259 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679272 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.679278 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.680178 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.680202 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.681195 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.681382 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.685325 2580 server.go:1268] "Started kubelet" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.685419 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.685513 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.685924 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbztm" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.685984 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.686038 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.687374 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.687695 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.690195 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.692202 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-136.ec2.internal.18c4a910bc0fcf66 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-136.ec2.internal,UID:ip-10-0-133-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-136.ec2.internal,},FirstTimestamp:2026-07-22 16:20:41.685282662 +0000 UTC m=+0.403278741,LastTimestamp:2026-07-22 16:20:41.685282662 +0000 UTC m=+0.403278741,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-136.ec2.internal,}" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.693701 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbztm" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.698553 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.698573 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.698693 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.699438 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.699508 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 16:20:42.201925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.699523 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.699652 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.699661 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.700985 2580 factory.go:55] Registering systemd factory Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701014 2580 factory.go:223] Registration of the systemd container factory successfully Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701210 2580 factory.go:153] Registering CRI-O factory Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701218 2580 factory.go:223] Registration of the crio container factory successfully Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701263 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701285 2580 factory.go:103] Registering Raw factory Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701295 2580 manager.go:1196] Started watching for new ooms in manager Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.701718 2580 manager.go:349] Starting recovery of all containers Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.708256 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.709828 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.711379 2580 manager.go:354] Recovery completed Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.711680 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-136.ec2.internal\" not found" node="ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.715909 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718099 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718125 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718134 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718633 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718642 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718653 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.718659 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.719251 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.719276 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.719285 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.720688 2580 policy_none.go:49] "None policy: Start" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.720701 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.720710 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.722661 2580 policy_none.go:47] "Start" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.799255 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.834858 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.834900 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.834922 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.834959 2580 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836241 2580 manager.go:337] "Starting Device Plugin manager" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.836267 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836277 2580 server.go:87] "Starting device plugin registration server" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836587 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836599 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836683 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836773 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.836782 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.837155 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.837206 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.837367 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.936000 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal"] Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.936105 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.937376 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.202970 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938821 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938841 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938852 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938863 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938868 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938878 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.938897 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.941274 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.941366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.941395 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.941978 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.941997 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.942007 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.942089 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.942112 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.942120 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.944154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.944176 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.944870 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.944886 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.944897 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:41.951334 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.951354 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-136.ec2.internal\": node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.968163 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-136.ec2.internal\" not found" node="ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.973097 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:41.973211 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-136.ec2.internal\" not found" node="ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.001598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07ab7404794cdf5e999aa34de5628077-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal\" (UID: \"07ab7404794cdf5e999aa34de5628077\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.001861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07ab7404794cdf5e999aa34de5628077-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal\" (UID: \"07ab7404794cdf5e999aa34de5628077\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.001893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/150be6474fc5a2b8d6c7faceb3ac3985-config\") pod \"kube-apiserver-proxy-ip-10-0-133-136.ec2.internal\" (UID: \"150be6474fc5a2b8d6c7faceb3ac3985\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:42.073150 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.204344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.102519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07ab7404794cdf5e999aa34de5628077-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal\" (UID: \"07ab7404794cdf5e999aa34de5628077\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.205147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.102550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07ab7404794cdf5e999aa34de5628077-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal\" (UID: \"07ab7404794cdf5e999aa34de5628077\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.205147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.102571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/150be6474fc5a2b8d6c7faceb3ac3985-config\") pod \"kube-apiserver-proxy-ip-10-0-133-136.ec2.internal\" (UID: \"150be6474fc5a2b8d6c7faceb3ac3985\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.205147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.102626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07ab7404794cdf5e999aa34de5628077-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal\" (UID: \"07ab7404794cdf5e999aa34de5628077\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.205147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.102637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07ab7404794cdf5e999aa34de5628077-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal\" (UID: \"07ab7404794cdf5e999aa34de5628077\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.205147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.102690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/150be6474fc5a2b8d6c7faceb3ac3985-config\") pod \"kube-apiserver-proxy-ip-10-0-133-136.ec2.internal\" (UID: \"150be6474fc5a2b8d6c7faceb3ac3985\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.205147 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:42.173853 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.392868 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.271322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.392868 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:42.274420 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.392868 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.276072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:42.392868 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:42.375298 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.551462 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:42.475836 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-136.ec2.internal\" not found" Jul 22 16:20:42.551462 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:42.550573 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.737331 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.751456 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.752775 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.757832 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.757937 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.757979 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:43.757989 2580 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a6377659f474841479464a2e7a8b6df1-e73ec76ff3f627c4.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.133.136:36300->100.57.222.105:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.757981 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.809161 2580 apiserver.go:52] "Watching apiserver" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.821914 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.822850 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sq52k","openshift-cluster-node-tuning-operator/tuned-mglmr","openshift-dns/node-resolver-hmmv2","openshift-multus/multus-9trzg","openshift-multus/multus-additional-cni-plugins-grhzz","openshift-network-operator/iptables-alerter-xrd7f","openshift-ovn-kubernetes/ovnkube-node-n2cgh","kube-system/global-pull-secret-syncer-l2gk6","kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j","openshift-image-registry/node-ca-dqpjx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal","openshift-multus/network-metrics-daemon-kfn8q","openshift-network-diagnostics/network-check-target-z6s8g"] Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.825457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.827646 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s8td2\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.827735 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.827933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.828127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.829838 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.829908 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.831709 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rgdbb\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.834956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.836716 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.836831 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.837027 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.837079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wz42t\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.837138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.837309 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 16:15:42 +0000 UTC" deadline="2028-01-25 03:59:57.543628294 +0000 UTC" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.837353 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13235h39m13.706279588s" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.838884 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.838931 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.839117 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.858337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.839211 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z5kfv\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.839233 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.839449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.841308 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.841323 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.841627 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f46vz\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.841737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-etc-kubernetes\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0a79340-0d73-44bb-9205-2a863050ec2a-agent-certs\") pod \"konnectivity-agent-sq52k\" (UID: \"f0a79340-0d73-44bb-9205-2a863050ec2a\") " pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-kubernetes\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-run\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-cnibin\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-socket-dir-parent\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-cni-bin\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-hostroot\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0a79340-0d73-44bb-9205-2a863050ec2a-konnectivity-ca\") pod \"konnectivity-agent-sq52k\" (UID: \"f0a79340-0d73-44bb-9205-2a863050ec2a\") " pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-lib-modules\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8zg\" (UniqueName: \"kubernetes.io/projected/98f66b90-9820-4888-99d0-b60a3d552c22-kube-api-access-zb8zg\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843926 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7qlp\"" Jul 22 16:20:43.860064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-system-cni-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.843992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-cni-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844013 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfjf\" (UniqueName: \"kubernetes.io/projected/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-kube-api-access-rgfjf\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-os-release\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysconfig\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysctl-d\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysctl-conf\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-var-lib-kubelet\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e375cc0-38fb-4257-8a70-bb24353d6f64-tmp-dir\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-netns\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-system-cni-dir\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-modprobe-d\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e375cc0-38fb-4257-8a70-bb24353d6f64-hosts-file\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-cni-multus\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-host\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.861502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-os-release\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2qr\" (UniqueName: \"kubernetes.io/projected/e27921ef-aaa0-4ef2-9380-99c6d21574e2-kube-api-access-xb2qr\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-sys\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98f66b90-9820-4888-99d0-b60a3d552c22-etc-tuned\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktmz\" (UniqueName: \"kubernetes.io/projected/8e375cc0-38fb-4257-8a70-bb24353d6f64-kube-api-access-jktmz\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-daemon-config\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-multus-certs\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cnibin\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-systemd\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98f66b90-9820-4888-99d0-b60a3d552c22-tmp\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-cni-binary-copy\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-k8s-cni-cncf-io\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-kubelet\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.844819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-conf-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.845874 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.846136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.848209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.864669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.850383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:43.846192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.857480 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.857507 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.857711 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.857733 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.857820 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6rhk\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.857864 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.858168 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.858238 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.858501 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.858687 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.858707 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.859273 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-whb5w\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.859520 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltrq5\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.859727 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.860265 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.860534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:43.860658 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.864426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:43.865501 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:43.864492 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.869507 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wkk5h" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.878728 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wkk5h" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.938235 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.944982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-cni-multus\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskvv\" (UniqueName: \"kubernetes.io/projected/512f6824-6869-48d9-88a4-c50435ebcb62-kube-api-access-sskvv\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/826c741a-501c-4c72-a2e9-a793749a666f-host\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h762\" (UniqueName: \"kubernetes.io/projected/826c741a-501c-4c72-a2e9-a793749a666f-kube-api-access-9h762\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-cni-multus\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-host\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-host\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/269497b7-5123-4d80-8735-471d64f1e566-kubelet-config\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/826c741a-501c-4c72-a2e9-a793749a666f-serviceca\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2qr\" (UniqueName: \"kubernetes.io/projected/e27921ef-aaa0-4ef2-9380-99c6d21574e2-kube-api-access-xb2qr\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-ovnkube-script-lib\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-sys\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.977109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktmz\" (UniqueName: \"kubernetes.io/projected/8e375cc0-38fb-4257-8a70-bb24353d6f64-kube-api-access-jktmz\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-daemon-config\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-systemd\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-ovnkube-config\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/269497b7-5123-4d80-8735-471d64f1e566-dbus\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-kubelet\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-sys\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-conf-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945694 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-conf-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/512f6824-6869-48d9-88a4-c50435ebcb62-iptables-alerter-script\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-node-log\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-kubelet\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-cni-bin\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-env-overrides\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-socket-dir-parent\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.978038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-hostroot\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-sys-fs\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.945990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-kubelet\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-hostroot\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-socket-dir-parent\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-ovn\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-log-socket\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-daemon-config\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0a79340-0d73-44bb-9205-2a863050ec2a-konnectivity-ca\") pod \"konnectivity-agent-sq52k\" (UID: \"f0a79340-0d73-44bb-9205-2a863050ec2a\") " pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-system-cni-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfjf\" (UniqueName: \"kubernetes.io/projected/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-kube-api-access-rgfjf\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-os-release\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946164 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-system-cni-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-os-release\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-socket-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.984820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysconfig\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e375cc0-38fb-4257-8a70-bb24353d6f64-tmp-dir\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysconfig\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-device-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fm5q\" (UniqueName: \"kubernetes.io/projected/33563127-f56b-4d29-bf78-451181ea866d-kube-api-access-5fm5q\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-modprobe-d\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e375cc0-38fb-4257-8a70-bb24353d6f64-hosts-file\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e375cc0-38fb-4257-8a70-bb24353d6f64-hosts-file\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqlf\" (UniqueName: \"kubernetes.io/projected/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-kube-api-access-cxqlf\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-cni-netd\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33563127-f56b-4d29-bf78-451181ea866d-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-modprobe-d\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-os-release\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-os-release\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/512f6824-6869-48d9-88a4-c50435ebcb62-host-slash\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-systemd-units\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.985671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e375cc0-38fb-4257-8a70-bb24353d6f64-tmp-dir\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0a79340-0d73-44bb-9205-2a863050ec2a-konnectivity-ca\") pod \"konnectivity-agent-sq52k\" (UID: \"f0a79340-0d73-44bb-9205-2a863050ec2a\") " pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-etc-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98f66b90-9820-4888-99d0-b60a3d552c22-etc-tuned\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-multus-certs\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cnibin\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-multus-certs\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-slash\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-var-lib-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-systemd\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98f66b90-9820-4888-99d0-b60a3d552c22-tmp\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-cni-binary-copy\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cnibin\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.986730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-k8s-cni-cncf-io\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-systemd\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946964 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-etc-kubernetes\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.946996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-k8s-cni-cncf-io\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-etc-kubernetes\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-run-netns\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0a79340-0d73-44bb-9205-2a863050ec2a-agent-certs\") pod \"konnectivity-agent-sq52k\" (UID: \"f0a79340-0d73-44bb-9205-2a863050ec2a\") " pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-kubernetes\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-run\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-cnibin\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-cni-bin\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-run\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-lib-modules\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-cni-binary-copy\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8zg\" (UniqueName: \"kubernetes.io/projected/98f66b90-9820-4888-99d0-b60a3d552c22-kube-api-access-zb8zg\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.987790 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947419 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-var-lib-cni-bin\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-cni-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-registration-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysctl-d\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysctl-conf\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-multus-cni-dir\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-var-lib-kubelet\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-netns\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-system-cni-dir\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxvg\" (UniqueName: \"kubernetes.io/projected/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-kube-api-access-lxxvg\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysctl-d\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-host-run-netns\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-var-lib-kubelet\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-sysctl-conf\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-cnibin\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-lib-modules\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.988851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e27921ef-aaa0-4ef2-9380-99c6d21574e2-system-cni-dir\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e27921ef-aaa0-4ef2-9380-99c6d21574e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.947816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98f66b90-9820-4888-99d0-b60a3d552c22-etc-kubernetes\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.950027 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98f66b90-9820-4888-99d0-b60a3d552c22-etc-tuned\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.950058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98f66b90-9820-4888-99d0-b60a3d552c22-tmp\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.950218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0a79340-0d73-44bb-9205-2a863050ec2a-agent-certs\") pod \"konnectivity-agent-sq52k\" (UID: \"f0a79340-0d73-44bb-9205-2a863050ec2a\") " pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.953466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2qr\" (UniqueName: \"kubernetes.io/projected/e27921ef-aaa0-4ef2-9380-99c6d21574e2-kube-api-access-xb2qr\") pod \"multus-additional-cni-plugins-grhzz\" (UID: \"e27921ef-aaa0-4ef2-9380-99c6d21574e2\") " pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.953676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktmz\" (UniqueName: \"kubernetes.io/projected/8e375cc0-38fb-4257-8a70-bb24353d6f64-kube-api-access-jktmz\") pod \"node-resolver-hmmv2\" (UID: \"8e375cc0-38fb-4257-8a70-bb24353d6f64\") " pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.953873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfjf\" (UniqueName: \"kubernetes.io/projected/e2aa321e-0ea9-4e99-b4b3-e1c80e538302-kube-api-access-rgfjf\") pod \"multus-9trzg\" (UID: \"e2aa321e-0ea9-4e99-b4b3-e1c80e538302\") " pod="openshift-multus/multus-9trzg" Jul 22 16:20:43.989665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:43.954660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8zg\" (UniqueName: \"kubernetes.io/projected/98f66b90-9820-4888-99d0-b60a3d552c22-kube-api-access-zb8zg\") pod \"tuned-mglmr\" (UID: \"98f66b90-9820-4888-99d0-b60a3d552c22\") " pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-cni-bin\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-env-overrides\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-sys-fs\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-kubelet\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-cni-bin\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.048970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-sys-fs\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-kubelet\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-ovn\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-log-socket\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-socket-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-ovn\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-device-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-log-socket\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-device-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm5q\" (UniqueName: \"kubernetes.io/projected/33563127-f56b-4d29-bf78-451181ea866d-kube-api-access-5fm5q\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.090335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqlf\" (UniqueName: \"kubernetes.io/projected/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-kube-api-access-cxqlf\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049294 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-socket-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-cni-netd\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33563127-f56b-4d29-bf78-451181ea866d-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-cni-netd\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-env-overrides\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/512f6824-6869-48d9-88a4-c50435ebcb62-host-slash\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-systemd-units\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.049455 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.049507 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:20:44.549487575 +0000 UTC m=+2.129581450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-systemd-units\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/512f6824-6869-48d9-88a4-c50435ebcb62-host-slash\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.049557 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-etc-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.091602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-slash\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-slash\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.049616 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:44.549598686 +0000 UTC m=+2.129692563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-etc-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-var-lib-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-var-lib-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-run-netns\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-registration-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxvg\" (UniqueName: \"kubernetes.io/projected/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-kube-api-access-lxxvg\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-run-netns\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskvv\" (UniqueName: \"kubernetes.io/projected/512f6824-6869-48d9-88a4-c50435ebcb62-kube-api-access-sskvv\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/826c741a-501c-4c72-a2e9-a793749a666f-host\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-registration-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h762\" (UniqueName: \"kubernetes.io/projected/826c741a-501c-4c72-a2e9-a793749a666f-kube-api-access-9h762\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.092238 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/826c741a-501c-4c72-a2e9-a793749a666f-host\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/269497b7-5123-4d80-8735-471d64f1e566-kubelet-config\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049917 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-openvswitch\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/826c741a-501c-4c72-a2e9-a793749a666f-serviceca\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/269497b7-5123-4d80-8735-471d64f1e566-kubelet-config\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.049995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-ovnkube-script-lib\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-systemd\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-ovnkube-config\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/269497b7-5123-4d80-8735-471d64f1e566-dbus\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/269497b7-5123-4d80-8735-471d64f1e566-dbus\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-run-systemd\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/512f6824-6869-48d9-88a4-c50435ebcb62-iptables-alerter-script\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.093065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-node-log\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33563127-f56b-4d29-bf78-451181ea866d-node-log\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-ovnkube-script-lib\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/512f6824-6869-48d9-88a4-c50435ebcb62-iptables-alerter-script\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33563127-f56b-4d29-bf78-451181ea866d-ovnkube-config\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.050732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/826c741a-501c-4c72-a2e9-a793749a666f-serviceca\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.051979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33563127-f56b-4d29-bf78-451181ea866d-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.057853 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.057875 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.057888 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.057956 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:44.557939509 +0000 UTC m=+2.138033397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.059813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqlf\" (UniqueName: \"kubernetes.io/projected/147aaecd-fc1d-46c6-8027-2a5a21ecff4e-kube-api-access-cxqlf\") pod \"aws-ebs-csi-driver-node-rmb6j\" (UID: \"147aaecd-fc1d-46c6-8027-2a5a21ecff4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.062339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxvg\" (UniqueName: \"kubernetes.io/projected/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-kube-api-access-lxxvg\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.062471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm5q\" (UniqueName: \"kubernetes.io/projected/33563127-f56b-4d29-bf78-451181ea866d-kube-api-access-5fm5q\") pod \"ovnkube-node-n2cgh\" (UID: \"33563127-f56b-4d29-bf78-451181ea866d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.062680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskvv\" (UniqueName: \"kubernetes.io/projected/512f6824-6869-48d9-88a4-c50435ebcb62-kube-api-access-sskvv\") pod \"iptables-alerter-xrd7f\" (UID: \"512f6824-6869-48d9-88a4-c50435ebcb62\") " pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.093707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.063027 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h762\" (UniqueName: \"kubernetes.io/projected/826c741a-501c-4c72-a2e9-a793749a666f-kube-api-access-9h762\") pod \"node-ca-dqpjx\" (UID: \"826c741a-501c-4c72-a2e9-a793749a666f\") " pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.182696 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.148096 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150be6474fc5a2b8d6c7faceb3ac3985.slice/crio-5d9b8e6ee6d80a89a932e99e6fc3c3dc1ae3d0551b57d7cee74e688749fe3649 WatchSource:0}: Error finding container 5d9b8e6ee6d80a89a932e99e6fc3c3dc1ae3d0551b57d7cee74e688749fe3649: Status 404 returned error can't find the container with id 5d9b8e6ee6d80a89a932e99e6fc3c3dc1ae3d0551b57d7cee74e688749fe3649 Jul 22 16:20:44.182696 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.152990 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:20:44.182696 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.157837 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ab7404794cdf5e999aa34de5628077.slice/crio-a91bc72c0600ff7adea4f2e3f63de82af0e22f4c1b04eae171de9afcf12677b5 WatchSource:0}: Error finding container a91bc72c0600ff7adea4f2e3f63de82af0e22f4c1b04eae171de9afcf12677b5: Status 404 returned error can't find the container with id a91bc72c0600ff7adea4f2e3f63de82af0e22f4c1b04eae171de9afcf12677b5 Jul 22 16:20:44.182696 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.172664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:20:44.182696 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.177602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mglmr" Jul 22 16:20:44.182696 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.181111 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a79340_0d73_44bb_9205_2a863050ec2a.slice/crio-fa1143fa5bd2aca863d13a1eb59bb1fcb026e490f3449ff7ad37129fa2ab0153 WatchSource:0}: Error finding container fa1143fa5bd2aca863d13a1eb59bb1fcb026e490f3449ff7ad37129fa2ab0153: Status 404 returned error can't find the container with id fa1143fa5bd2aca863d13a1eb59bb1fcb026e490f3449ff7ad37129fa2ab0153 Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.189376 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f66b90_9820_4888_99d0_b60a3d552c22.slice/crio-3d24288bb18f78d7944a75e2d2d5c66f35ff2cf405961ef6b667ed7026e01208 WatchSource:0}: Error finding container 3d24288bb18f78d7944a75e2d2d5c66f35ff2cf405961ef6b667ed7026e01208: Status 404 returned error can't find the container with id 3d24288bb18f78d7944a75e2d2d5c66f35ff2cf405961ef6b667ed7026e01208 Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.199481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmmv2" Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.207243 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e375cc0_38fb_4257_8a70_bb24353d6f64.slice/crio-3213db1791e88ef6a76d30fb8d26b46d8bf2461c52ec870556ae98263e94bcf1 WatchSource:0}: Error finding container 3213db1791e88ef6a76d30fb8d26b46d8bf2461c52ec870556ae98263e94bcf1: Status 404 returned error can't find the container with id 3213db1791e88ef6a76d30fb8d26b46d8bf2461c52ec870556ae98263e94bcf1 Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.214961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9trzg" Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.222881 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2aa321e_0ea9_4e99_b4b3_e1c80e538302.slice/crio-dd160ecdc98494c42e2fa830767f79590b2526aaebf13bb6f3984b05a41a67ba WatchSource:0}: Error finding container dd160ecdc98494c42e2fa830767f79590b2526aaebf13bb6f3984b05a41a67ba: Status 404 returned error can't find the container with id dd160ecdc98494c42e2fa830767f79590b2526aaebf13bb6f3984b05a41a67ba Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.241581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grhzz" Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.250129 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27921ef_aaa0_4ef2_9380_99c6d21574e2.slice/crio-59180c70f6ccf4af33116fbd18e3fed6301e67b050d41a78a8aa492d482c6a73 WatchSource:0}: Error finding container 59180c70f6ccf4af33116fbd18e3fed6301e67b050d41a78a8aa492d482c6a73: Status 404 returned error can't find the container with id 59180c70f6ccf4af33116fbd18e3fed6301e67b050d41a78a8aa492d482c6a73 Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.290430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xrd7f" Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.298430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:20:44.306105 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.298650 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512f6824_6869_48d9_88a4_c50435ebcb62.slice/crio-b7c47a76a39371449d388383bb9f91e357ec6a2a6f5bd062e1863577beb06a47 WatchSource:0}: Error finding container b7c47a76a39371449d388383bb9f91e357ec6a2a6f5bd062e1863577beb06a47: Status 404 returned error can't find the container with id b7c47a76a39371449d388383bb9f91e357ec6a2a6f5bd062e1863577beb06a47 Jul 22 16:20:44.307761 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.306170 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33563127_f56b_4d29_bf78_451181ea866d.slice/crio-9532c1517fffd78a3b5a382ca7ce9316b6f61015ede1a7e29f5bd0766e9aa385 WatchSource:0}: Error finding container 9532c1517fffd78a3b5a382ca7ce9316b6f61015ede1a7e29f5bd0766e9aa385: Status 404 returned error can't find the container with id 9532c1517fffd78a3b5a382ca7ce9316b6f61015ede1a7e29f5bd0766e9aa385 Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.329457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.332435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqpjx" Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.338868 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147aaecd_fc1d_46c6_8027_2a5a21ecff4e.slice/crio-11d9193a8dcdd872a4fbb47d429e71f57a7e8fa4e915d85789d9427fbfb0f900 WatchSource:0}: Error finding container 11d9193a8dcdd872a4fbb47d429e71f57a7e8fa4e915d85789d9427fbfb0f900: Status 404 returned error can't find the container with id 11d9193a8dcdd872a4fbb47d429e71f57a7e8fa4e915d85789d9427fbfb0f900 Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:20:44.341028 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826c741a_501c_4c72_a2e9_a793749a666f.slice/crio-b3e5f680069b352138f8f81dc5f9faadd548422505fa40710409f473fdf0739b WatchSource:0}: Error finding container b3e5f680069b352138f8f81dc5f9faadd548422505fa40710409f473fdf0739b: Status 404 returned error can't find the container with id b3e5f680069b352138f8f81dc5f9faadd548422505fa40710409f473fdf0739b Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.344065 2580 kuberuntime_manager.go:1459] "Unhandled Error" err="container node-ca start failed in pod node-ca-dqpjx_openshift-image-registry(826c741a-501c-4c72-a2e9-a793749a666f): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.344090 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-image-registry/node-ca-dqpjx" podUID="826c741a-501c-4c72-a2e9-a793749a666f" Jul 22 16:20:44.430823 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.373741 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:44.555905 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.554118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:44.555905 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.554149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.555905 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.554246 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:44.555905 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.554255 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:44.555905 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.554296 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:45.554283076 +0000 UTC m=+3.134376951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:44.555905 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.554310 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:20:45.554304393 +0000 UTC m=+3.134398268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:44.680810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.655415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:44.680810 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.655636 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:20:44.680810 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.655657 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:20:44.680810 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.655670 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:44.680810 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.655725 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:45.655707771 +0000 UTC m=+3.235801648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:44.858161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.854611 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.881344 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:15:43 +0000 UTC" deadline="2028-02-16 22:32:57.598687035 +0000 UTC" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.881412 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13782h12m12.717297419s" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.974517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.974648 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.974696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:44.974824 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.980260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xrd7f" event={"ID":"512f6824-6869-48d9-88a4-c50435ebcb62","Type":"ContainerStarted","Data":"b7c47a76a39371449d388383bb9f91e357ec6a2a6f5bd062e1863577beb06a47"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.982139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerStarted","Data":"59180c70f6ccf4af33116fbd18e3fed6301e67b050d41a78a8aa492d482c6a73"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.983858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9trzg" event={"ID":"e2aa321e-0ea9-4e99-b4b3-e1c80e538302","Type":"ContainerStarted","Data":"dd160ecdc98494c42e2fa830767f79590b2526aaebf13bb6f3984b05a41a67ba"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.986080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mglmr" event={"ID":"98f66b90-9820-4888-99d0-b60a3d552c22","Type":"ContainerStarted","Data":"3d24288bb18f78d7944a75e2d2d5c66f35ff2cf405961ef6b667ed7026e01208"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.987611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sq52k" event={"ID":"f0a79340-0d73-44bb-9205-2a863050ec2a","Type":"ContainerStarted","Data":"fa1143fa5bd2aca863d13a1eb59bb1fcb026e490f3449ff7ad37129fa2ab0153"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.989871 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" event={"ID":"147aaecd-fc1d-46c6-8027-2a5a21ecff4e","Type":"ContainerStarted","Data":"11d9193a8dcdd872a4fbb47d429e71f57a7e8fa4e915d85789d9427fbfb0f900"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.992378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"9532c1517fffd78a3b5a382ca7ce9316b6f61015ede1a7e29f5bd0766e9aa385"} Jul 22 16:20:44.994764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.994122 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmmv2" event={"ID":"8e375cc0-38fb-4257-8a70-bb24353d6f64","Type":"ContainerStarted","Data":"3213db1791e88ef6a76d30fb8d26b46d8bf2461c52ec870556ae98263e94bcf1"} Jul 22 16:20:44.999018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.996350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" event={"ID":"07ab7404794cdf5e999aa34de5628077","Type":"ContainerStarted","Data":"a91bc72c0600ff7adea4f2e3f63de82af0e22f4c1b04eae171de9afcf12677b5"} Jul 22 16:20:44.999018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:44.998859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" event={"ID":"150be6474fc5a2b8d6c7faceb3ac3985","Type":"ContainerStarted","Data":"5d9b8e6ee6d80a89a932e99e6fc3c3dc1ae3d0551b57d7cee74e688749fe3649"} Jul 22 16:20:45.007115 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.000523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqpjx" event={"ID":"826c741a-501c-4c72-a2e9-a793749a666f","Type":"ContainerStarted","Data":"b3e5f680069b352138f8f81dc5f9faadd548422505fa40710409f473fdf0739b"} Jul 22 16:20:45.007115 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.002181 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b8c58be4906c5077240bfb7d1315ca7709d0915b03b2b7020d68d6959102cf8\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-image-registry/node-ca-dqpjx" podUID="826c741a-501c-4c72-a2e9-a793749a666f" Jul 22 16:20:45.291832 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.291020 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:45.568115 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.568027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:45.568115 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.568072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:45.568314 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.568237 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:45.568314 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.568293 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:47.568273997 +0000 UTC m=+5.148367895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:45.568522 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.568506 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:45.568583 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.568555 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:20:47.568541721 +0000 UTC m=+5.148635610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:45.668950 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.668459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:45.668950 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.668667 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:20:45.668950 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.668687 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:20:45.668950 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.668701 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:45.668950 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.668769 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:47.668749953 +0000 UTC m=+5.248843841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:45.882430 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.882319 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:15:43 +0000 UTC" deadline="2028-01-25 10:02:58.612225533 +0000 UTC" Jul 22 16:20:45.882430 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.882365 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13241h42m12.729864863s" Jul 22 16:20:45.973132 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:45.973014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:45.973292 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:45.973148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:46.755074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:46.754835 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:20:46.974963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:46.974891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:46.975160 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:46.975028 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:46.975582 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:46.975561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:46.975691 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:46.975670 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:47.584987 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:47.584931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:47.585153 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:47.585005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:47.585153 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.585132 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:47.585270 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.585196 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:51.585175981 +0000 UTC m=+9.165269857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:47.585692 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.585667 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:47.585799 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.585754 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:20:51.585713212 +0000 UTC m=+9.165807088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:47.685882 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:47.685818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:47.686062 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.686044 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:20:47.686137 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.686074 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:20:47.686137 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.686088 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:47.686235 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.686154 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:51.686132894 +0000 UTC m=+9.266226774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:47.973943 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:47.973352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:47.973943 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:47.973571 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:48.973679 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:48.973641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:48.973862 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:48.973771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:48.973862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:48.973829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:48.973972 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:48.973943 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:49.974054 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:49.973483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:49.974054 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:49.973630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:50.973430 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:50.973382 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:50.973594 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:50.973533 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:50.973938 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:50.973907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:50.974042 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:50.974015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:51.622189 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:51.621846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:51.622189 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:51.621891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:51.622189 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.622010 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:51.622189 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.622064 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:59.622046373 +0000 UTC m=+17.202140264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:51.622189 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.622126 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:51.622189 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.622190 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:20:59.622175529 +0000 UTC m=+17.202269409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:51.722423 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:51.722333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:51.722588 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.722531 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:20:51.722588 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.722552 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:20:51.722588 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.722565 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:51.722740 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.722620 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:20:59.722602787 +0000 UTC m=+17.302696680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:51.973991 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:51.973903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:51.974160 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:51.974030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:52.974885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:52.974851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:52.975343 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:52.974963 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:52.975343 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:52.975027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:52.975343 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:52.975117 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:53.973124 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:53.973093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:53.973271 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:53.973239 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:54.973583 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:54.973549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:54.974069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:54.973564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:54.974069 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:54.973678 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:54.974069 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:54.973750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:55.973427 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:55.973376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:55.973611 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:55.973531 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:56.973069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:56.973032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:56.973239 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:56.973038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:56.973239 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:56.973146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:56.973358 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:56.973246 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:57.973924 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:57.973886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:57.974379 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:57.974024 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:20:58.973946 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:58.973903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:58.974302 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:58.973954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:58.974302 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:58.974049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:20:58.974302 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:58.974123 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:20:59.679370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:59.679325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:59.679370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:59.679377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:20:59.679666 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.679511 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:59.679666 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.679581 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:15.679561673 +0000 UTC m=+33.259655564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:20:59.679666 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.679511 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:59.679818 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.679689 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:21:15.679666267 +0000 UTC m=+33.259760146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:20:59.779931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:59.779878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:20:59.780089 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.780056 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:20:59.780089 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.780074 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:20:59.780089 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.780083 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:59.780234 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.780137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:15.780120501 +0000 UTC m=+33.360214379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:20:59.973765 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:20:59.973690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:20:59.973929 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:20:59.973811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:00.973454 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:00.973421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:00.973905 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:00.973543 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:00.973905 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:00.973600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:00.973905 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:00.973702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:01.973030 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:01.972997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:01.973227 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:01.973132 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:02.974040 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:02.974007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:02.974439 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:02.974109 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:02.974439 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:02.974149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:02.974439 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:02.974196 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:03.974094 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:03.973741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:03.974807 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:03.974214 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:04.043325 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.043296 2580 generic.go:356] "Generic (PLEG): container finished" podID="e27921ef-aaa0-4ef2-9380-99c6d21574e2" containerID="584bb734e48ee2774de5e92de66e812ab4665479336b70d8e4622d576f6560ed" exitCode=0 Jul 22 16:21:04.043453 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.043387 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerDied","Data":"584bb734e48ee2774de5e92de66e812ab4665479336b70d8e4622d576f6560ed"} Jul 22 16:21:04.045079 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.045056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9trzg" event={"ID":"e2aa321e-0ea9-4e99-b4b3-e1c80e538302","Type":"ContainerStarted","Data":"923f8ca8dcf02bdf52433afc201c858ecae44e1b287c6c0a664904b00ab53004"} Jul 22 16:21:04.046720 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.046660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mglmr" event={"ID":"98f66b90-9820-4888-99d0-b60a3d552c22","Type":"ContainerStarted","Data":"55a654aaab35930a3b068f463092c885db8122cc1faa848c8d3e7bf929bc8bf0"} Jul 22 16:21:04.049124 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.048670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sq52k" event={"ID":"f0a79340-0d73-44bb-9205-2a863050ec2a","Type":"ContainerStarted","Data":"17040c1da31c3b665080fea53cf22224c42192bdb88889526848d2834c911ecd"} Jul 22 16:21:04.054912 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.054889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" event={"ID":"147aaecd-fc1d-46c6-8027-2a5a21ecff4e","Type":"ContainerStarted","Data":"f4bfde81885391594506709b4a50e291bd9f1cfadb721aac52b2f44281fc5d90"} Jul 22 16:21:04.057638 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.057606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"d1ec26eab5ffcf314e899338f0bc4a83d84e9f868b7880e25e8bbcdb973e89c9"} Jul 22 16:21:04.057724 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.057668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"15c9175503e3826c1b121ba3979f580099ac2b6441f765917a1c2d9912537d21"} Jul 22 16:21:04.057724 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.057698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"c1ae1c64156a7ef83b7d06da15e682dff5a770f57e56dddb2a176ee2068712eb"} Jul 22 16:21:04.057724 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.057725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"80866af9c5ca91baec0c6276a03bc385a9ba3287a574727098b1fc821a979b4d"} Jul 22 16:21:04.059026 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.059004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmmv2" event={"ID":"8e375cc0-38fb-4257-8a70-bb24353d6f64","Type":"ContainerStarted","Data":"a84875113834dce05ea5a16308da6146ae4e5ee7dcdf53d7c236f6d918d56bd4"} Jul 22 16:21:04.060707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.060686 2580 generic.go:356] "Generic (PLEG): container finished" podID="07ab7404794cdf5e999aa34de5628077" containerID="f7789f9fce8d4d707a8d0999ef22ee726a97a0081ffc8d97109b63ddce23ed6f" exitCode=0 Jul 22 16:21:04.060795 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.060760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" event={"ID":"07ab7404794cdf5e999aa34de5628077","Type":"ContainerDied","Data":"f7789f9fce8d4d707a8d0999ef22ee726a97a0081ffc8d97109b63ddce23ed6f"} Jul 22 16:21:04.062476 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.062454 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" event={"ID":"150be6474fc5a2b8d6c7faceb3ac3985","Type":"ContainerStarted","Data":"f6e633e6d50b928e82658ae0e2b8a5bae5e7fc34cd54a24d141a86f57241c112"} Jul 22 16:21:04.088934 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.088882 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-136.ec2.internal" podStartSLOduration=21.088862078 podStartE2EDuration="21.088862078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:04.087868651 +0000 UTC m=+21.667962551" watchObservedRunningTime="2026-07-22 16:21:04.088862078 +0000 UTC m=+21.668955968" Jul 22 16:21:04.103925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.103876 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sq52k" podStartSLOduration=11.352277874 podStartE2EDuration="21.103859641s" totalImagesPullingTime="9.751581767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:04.103181764 +0000 UTC m=+21.683275682" watchObservedRunningTime="2026-07-22 16:21:04.103859641 +0000 UTC m=+21.683953540" Jul 22 16:21:04.120311 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.120257 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mglmr" podStartSLOduration=2.138038367 podStartE2EDuration="21.120238682s" totalImagesPullingTime="18.982200315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:04.119821901 +0000 UTC m=+21.699915797" watchObservedRunningTime="2026-07-22 16:21:04.120238682 +0000 UTC m=+21.700332580" Jul 22 16:21:04.135456 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.135415 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmmv2" podStartSLOduration=2.180143228 podStartE2EDuration="21.135383879s" totalImagesPullingTime="18.955240651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:04.135046881 +0000 UTC m=+21.715140779" watchObservedRunningTime="2026-07-22 16:21:04.135383879 +0000 UTC m=+21.715477775" Jul 22 16:21:04.152636 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.152558 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9trzg" podStartSLOduration=2.171619401 podStartE2EDuration="21.152539914s" totalImagesPullingTime="18.980920513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:04.152115432 +0000 UTC m=+21.732209328" watchObservedRunningTime="2026-07-22 16:21:04.152539914 +0000 UTC m=+21.732633811" Jul 22 16:21:04.977181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.976956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:04.977739 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:04.977010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:04.977739 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:04.977278 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:04.977739 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:04.977381 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:05.067876 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.067845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"756ebf04408975b9ea865a98752fc79f9de5d24488f0f15f8a03cfdd315cfb77"} Jul 22 16:21:05.068057 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.067924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"f7094c0f122b6b64969917a1ccc36b618b0202a14e0be319c4b1e0cd55371137"} Jul 22 16:21:05.069449 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.069421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqpjx" event={"ID":"826c741a-501c-4c72-a2e9-a793749a666f","Type":"ContainerStarted","Data":"9b847a04b873c1853c3d4e9a6e26d39d12247893d28ab4ca1c02f25853592004"} Jul 22 16:21:05.071007 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.070976 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xrd7f" event={"ID":"512f6824-6869-48d9-88a4-c50435ebcb62","Type":"ContainerStarted","Data":"effe8af414470e5ea86bec53aa6ab24f7169ea7ccdfc5bbb44310e05c48e8ee5"} Jul 22 16:21:05.086074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.086021 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dqpjx" podStartSLOduration=1.539411956 podStartE2EDuration="22.08600075s" totalImagesPullingTime="20.546588794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-22 16:21:05.085360544 +0000 UTC m=+22.665454442" watchObservedRunningTime="2026-07-22 16:21:05.08600075 +0000 UTC m=+22.666094652" Jul 22 16:21:05.117712 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.117648 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xrd7f" podStartSLOduration=3.253217666 podStartE2EDuration="22.117626717s" totalImagesPullingTime="18.864409051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:05.101068331 +0000 UTC m=+22.681162230" watchObservedRunningTime="2026-07-22 16:21:05.117626717 +0000 UTC m=+22.697720614" Jul 22 16:21:05.680660 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.680632 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 16:21:05.973318 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.973246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:05.973512 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:05.973359 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:05.981871 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.981782 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T16:21:05.680655138Z","UUID":"ee36a894-f019-4d3d-94b0-33687e971b42","Handler":null,"Name":"","Endpoint":""} Jul 22 16:21:05.983379 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.983352 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 16:21:05.983498 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:05.983411 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 16:21:06.075283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:06.075245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" event={"ID":"147aaecd-fc1d-46c6-8027-2a5a21ecff4e","Type":"ContainerStarted","Data":"5ff1fdf7ea3c71ab524401081edaab35435e7e6eb3409a17be44236cff0284e1"} Jul 22 16:21:06.077368 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:06.077336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" event={"ID":"07ab7404794cdf5e999aa34de5628077","Type":"ContainerStarted","Data":"a9fac8e6fe04cef552f3c9296eacea8690737149f88c53c78cd23a82662cb2bd"} Jul 22 16:21:06.973862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:06.973816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:06.974053 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:06.973818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:06.974053 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:06.973943 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:06.974053 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:06.974015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:07.082359 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.082323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" event={"ID":"147aaecd-fc1d-46c6-8027-2a5a21ecff4e","Type":"ContainerStarted","Data":"6cfac341abbc6cfc1fb810658045c1e657939e800d3667ff0b05fc4b3530ad6b"} Jul 22 16:21:07.086326 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.086301 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"07933a9c5021de621ecdaf2699213e9b7ec6ab39b1f6e0f99ab4f424c622bc6b"} Jul 22 16:21:07.101440 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.101375 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rmb6j" podStartSLOduration=2.278727918 podStartE2EDuration="24.101358506s" totalImagesPullingTime="21.822630588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:07.100807148 +0000 UTC m=+24.680901044" watchObservedRunningTime="2026-07-22 16:21:07.101358506 +0000 UTC m=+24.681452402" Jul 22 16:21:07.101629 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.101531 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-136.ec2.internal" podStartSLOduration=24.101522067 podStartE2EDuration="24.101522067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:06.09287629 +0000 UTC m=+23.672970197" watchObservedRunningTime="2026-07-22 16:21:07.101522067 +0000 UTC m=+24.681615966" Jul 22 16:21:07.223165 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.223090 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:21:07.223879 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.223857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:21:07.973596 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:07.973561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:07.973797 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:07.973685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:08.088474 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:08.088416 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:21:08.088933 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:08.088757 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sq52k" Jul 22 16:21:08.976338 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:08.976196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:08.976447 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:08.976246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:08.976447 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:08.976436 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:08.976519 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:08.976481 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:09.091725 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:09.091697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerStarted","Data":"6c9dcf7acdf7f3108915f8ab17e1ae479b12fbf1ae8584f6760c31b54dc320d7"} Jul 22 16:21:09.095344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:09.095252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" event={"ID":"33563127-f56b-4d29-bf78-451181ea866d","Type":"ContainerStarted","Data":"ccf6541b0ef0bee0f18d8f92442ffc37b7284aad9d02f67b46956d846b6ba6fb"} Jul 22 16:21:09.138046 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:09.138007 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" podStartSLOduration=6.92288359 podStartE2EDuration="26.137994045s" totalImagesPullingTime="19.215110455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:09.137235053 +0000 UTC m=+26.717328991" watchObservedRunningTime="2026-07-22 16:21:09.137994045 +0000 UTC m=+26.718087935" Jul 22 16:21:09.973784 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:09.973744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:09.974025 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:09.973859 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:10.098916 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.098882 2580 generic.go:356] "Generic (PLEG): container finished" podID="e27921ef-aaa0-4ef2-9380-99c6d21574e2" containerID="6c9dcf7acdf7f3108915f8ab17e1ae479b12fbf1ae8584f6760c31b54dc320d7" exitCode=0 Jul 22 16:21:10.099439 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.098952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerDied","Data":"6c9dcf7acdf7f3108915f8ab17e1ae479b12fbf1ae8584f6760c31b54dc320d7"} Jul 22 16:21:10.099439 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.099130 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:21:10.099439 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.099202 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:21:10.099690 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.099667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:21:10.114003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.113981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:21:10.114252 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.114236 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:21:10.683070 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.682886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z6s8g"] Jul 22 16:21:10.683224 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.683164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:10.683264 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:10.683244 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:10.685887 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.685839 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2gk6"] Jul 22 16:21:10.686148 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.685952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:10.686148 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:10.686058 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:10.686724 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.686701 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfn8q"] Jul 22 16:21:10.686824 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:10.686781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:10.686880 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:10.686859 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:11.102822 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:11.102792 2580 generic.go:356] "Generic (PLEG): container finished" podID="e27921ef-aaa0-4ef2-9380-99c6d21574e2" containerID="414da56dbf515b642e5a0192d765d8151662e5351851862f41dc55dbfa43fd83" exitCode=0 Jul 22 16:21:11.103420 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:11.102878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerDied","Data":"414da56dbf515b642e5a0192d765d8151662e5351851862f41dc55dbfa43fd83"} Jul 22 16:21:11.974039 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:11.974002 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:11.974039 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:11.974044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:11.974244 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:11.974095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:11.974244 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:11.974108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:11.974244 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:11.974172 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:11.974244 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:11.974228 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:12.765713 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:12.765679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:12.848864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:12.848830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:13.109827 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.109795 2580 generic.go:356] "Generic (PLEG): container finished" podID="e27921ef-aaa0-4ef2-9380-99c6d21574e2" containerID="f2084e6729c1aa5440e0032fb02fc703261283c075b96b12e45a6b67b86b0ebf" exitCode=0 Jul 22 16:21:13.110073 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.109852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerDied","Data":"f2084e6729c1aa5440e0032fb02fc703261283c075b96b12e45a6b67b86b0ebf"} Jul 22 16:21:13.299535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.299497 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:13.356538 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.356505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:13.974018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.973982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:13.974018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.974000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:13.974585 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:13.974150 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:13.974585 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:13.974212 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:13.974585 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:13.974265 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:13.974585 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:13.974359 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:15.697476 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:15.697437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:15.697933 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:15.697484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:15.697933 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.697623 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:21:15.697933 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.697624 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:21:15.697933 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.697681 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret podName:269497b7-5123-4d80-8735-471d64f1e566 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:47.697668445 +0000 UTC m=+65.277762324 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret") pod "global-pull-secret-syncer-l2gk6" (UID: "269497b7-5123-4d80-8735-471d64f1e566") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:21:15.697933 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.697699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:21:47.697690481 +0000 UTC m=+65.277784356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:21:15.798167 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:15.798123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:15.798368 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.798340 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:21:15.798463 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.798370 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:21:15.798463 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.798385 2580 projected.go:196] Error preparing data for projected volume kube-api-access-n629f for pod openshift-network-diagnostics/network-check-target-z6s8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:21:15.798550 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.798468 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f podName:f4123489-a5d6-4004-8315-36707890bd18 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:47.798449436 +0000 UTC m=+65.378543316 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n629f" (UniqueName: "kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f") pod "network-check-target-z6s8g" (UID: "f4123489-a5d6-4004-8315-36707890bd18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:21:15.973662 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:15.973622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:15.973829 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:15.973634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:15.973829 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.973760 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2gk6" podUID="269497b7-5123-4d80-8735-471d64f1e566" Jul 22 16:21:15.973829 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:15.973644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:15.973829 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.973814 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z6s8g" podUID="f4123489-a5d6-4004-8315-36707890bd18" Jul 22 16:21:15.973992 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:15.973881 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfn8q" podUID="d79635de-e4ce-4ebc-9921-b7e89ccd30bb" Jul 22 16:21:17.225673 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.225649 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-136.ec2.internal" event="NodeReady" Jul 22 16:21:17.226008 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.225801 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 16:21:17.271619 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.271588 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5b4ll"] Jul 22 16:21:17.274409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.274369 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.274409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.274383 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7x4pm"] Jul 22 16:21:17.276669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.276647 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 16:21:17.276788 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.276751 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 16:21:17.276788 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.276760 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4ntcb\"" Jul 22 16:21:17.277282 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.277267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.279128 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.279105 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 16:21:17.279229 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.279192 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 16:21:17.279290 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.279248 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 16:21:17.279290 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.279262 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4chrv\"" Jul 22 16:21:17.285317 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.285294 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5b4ll"] Jul 22 16:21:17.288346 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.288300 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7x4pm"] Jul 22 16:21:17.408675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.408643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrs7w\" (UniqueName: \"kubernetes.io/projected/21622b2f-0c96-4f74-bc80-832288568931-kube-api-access-vrs7w\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.408675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.408680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21622b2f-0c96-4f74-bc80-832288568931-config-volume\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.408880 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.408729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.408880 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.408778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5r8\" (UniqueName: \"kubernetes.io/projected/e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664-kube-api-access-mm5r8\") pod \"ingress-canary-7x4pm\" (UID: \"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664\") " pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.408880 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.408799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21622b2f-0c96-4f74-bc80-832288568931-tmp-dir\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.408880 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.408819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664-cert\") pod \"ingress-canary-7x4pm\" (UID: \"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664\") " pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.510065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.509986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrs7w\" (UniqueName: \"kubernetes.io/projected/21622b2f-0c96-4f74-bc80-832288568931-kube-api-access-vrs7w\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.510065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21622b2f-0c96-4f74-bc80-832288568931-config-volume\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.510065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.510263 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5r8\" (UniqueName: \"kubernetes.io/projected/e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664-kube-api-access-mm5r8\") pod \"ingress-canary-7x4pm\" (UID: \"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664\") " pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.510263 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21622b2f-0c96-4f74-bc80-832288568931-tmp-dir\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.510263 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664-cert\") pod \"ingress-canary-7x4pm\" (UID: \"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664\") " pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.510263 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:17.510157 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:21:17.510263 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:17.510238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls podName:21622b2f-0c96-4f74-bc80-832288568931 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:18.010216977 +0000 UTC m=+35.590310863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls") pod "dns-default-5b4ll" (UID: "21622b2f-0c96-4f74-bc80-832288568931") : secret "dns-default-metrics-tls" not found Jul 22 16:21:17.510535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21622b2f-0c96-4f74-bc80-832288568931-tmp-dir\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.510756 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.510734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21622b2f-0c96-4f74-bc80-832288568931-config-volume\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.514481 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.514463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664-cert\") pod \"ingress-canary-7x4pm\" (UID: \"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664\") " pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.521148 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.521126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrs7w\" (UniqueName: \"kubernetes.io/projected/21622b2f-0c96-4f74-bc80-832288568931-kube-api-access-vrs7w\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:17.521361 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.521345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5r8\" (UniqueName: \"kubernetes.io/projected/e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664-kube-api-access-mm5r8\") pod \"ingress-canary-7x4pm\" (UID: \"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664\") " pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.595481 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.595441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7x4pm" Jul 22 16:21:17.731311 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.731279 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7x4pm"] Jul 22 16:21:17.735536 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:17.735512 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05bcdb7_ae0b_48e3_b8e1_0b47afaa6664.slice/crio-cfa6db8dcc49bc0061354623cfef2aa69d5e664ab344dfb654a041cdd192ebb9 WatchSource:0}: Error finding container cfa6db8dcc49bc0061354623cfef2aa69d5e664ab344dfb654a041cdd192ebb9: Status 404 returned error can't find the container with id cfa6db8dcc49bc0061354623cfef2aa69d5e664ab344dfb654a041cdd192ebb9 Jul 22 16:21:17.973928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.973822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:17.974068 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.973822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:17.974068 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.973822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:17.978166 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.978143 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:21:17.978282 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.978167 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:21:17.978282 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.978218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:21:17.978282 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.978147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:21:17.978282 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.978145 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9mc4s\"" Jul 22 16:21:17.978463 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:17.978359 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6qdrm\"" Jul 22 16:21:18.013894 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:18.013863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:18.014055 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:18.013996 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:21:18.014109 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:18.014074 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls podName:21622b2f-0c96-4f74-bc80-832288568931 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:19.014052283 +0000 UTC m=+36.594146171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls") pod "dns-default-5b4ll" (UID: "21622b2f-0c96-4f74-bc80-832288568931") : secret "dns-default-metrics-tls" not found Jul 22 16:21:18.121974 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:18.121941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7x4pm" event={"ID":"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664","Type":"ContainerStarted","Data":"cfa6db8dcc49bc0061354623cfef2aa69d5e664ab344dfb654a041cdd192ebb9"} Jul 22 16:21:18.124439 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:18.124388 2580 generic.go:356] "Generic (PLEG): container finished" podID="e27921ef-aaa0-4ef2-9380-99c6d21574e2" containerID="9d29efdd3bcf89b04e3da9bf2a0819a2670e9a68da68677915bf1aaa1e668882" exitCode=0 Jul 22 16:21:18.124574 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:18.124439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerDied","Data":"9d29efdd3bcf89b04e3da9bf2a0819a2670e9a68da68677915bf1aaa1e668882"} Jul 22 16:21:19.021329 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:19.021121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:19.022074 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:19.021295 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:21:19.022074 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:19.021434 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls podName:21622b2f-0c96-4f74-bc80-832288568931 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:21.021415935 +0000 UTC m=+38.601509814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls") pod "dns-default-5b4ll" (UID: "21622b2f-0c96-4f74-bc80-832288568931") : secret "dns-default-metrics-tls" not found Jul 22 16:21:19.129714 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:19.129678 2580 generic.go:356] "Generic (PLEG): container finished" podID="e27921ef-aaa0-4ef2-9380-99c6d21574e2" containerID="33d58789704c3f06438e6fb17215943dd3f803c447554e772bf9423588e34762" exitCode=0 Jul 22 16:21:19.129898 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:19.129736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerDied","Data":"33d58789704c3f06438e6fb17215943dd3f803c447554e772bf9423588e34762"} Jul 22 16:21:20.135414 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.135370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grhzz" event={"ID":"e27921ef-aaa0-4ef2-9380-99c6d21574e2","Type":"ContainerStarted","Data":"4fc2b7b6deea6bb033e04bef47cda8b54479f9dc2df4f9f54ddfdfd6640af91c"} Jul 22 16:21:20.160229 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.160175 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-grhzz" podStartSLOduration=7.259373698 podStartE2EDuration="37.16015987s" totalImagesPullingTime="29.900786172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:20.159854403 +0000 UTC m=+37.739948299" watchObservedRunningTime="2026-07-22 16:21:20.16015987 +0000 UTC m=+37.740253767" Jul 22 16:21:20.286311 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.286277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:20.348216 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.348125 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:20.469838 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.469811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:20.571444 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.571415 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:20.870144 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.870108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:20.925315 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:20.925281 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:21.035229 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:21.035191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:21.035422 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:21.035356 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:21:21.035505 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:21.035447 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls podName:21622b2f-0c96-4f74-bc80-832288568931 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:25.035428147 +0000 UTC m=+42.615522039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls") pod "dns-default-5b4ll" (UID: "21622b2f-0c96-4f74-bc80-832288568931") : secret "dns-default-metrics-tls" not found Jul 22 16:21:21.139610 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:21.139528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7x4pm" event={"ID":"e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664","Type":"ContainerStarted","Data":"018eb563b0c746dbb6f5565fd10268d6526163885fa8d005b69a63868aa88fdd"} Jul 22 16:21:21.156168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:21.156114 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7x4pm" podStartSLOduration=1.8236217799999999 podStartE2EDuration="4.156099879s" totalImagesPullingTime="2.332478099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:21.155459302 +0000 UTC m=+38.735553198" watchObservedRunningTime="2026-07-22 16:21:21.156099879 +0000 UTC m=+38.736193829" Jul 22 16:21:25.065555 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:25.065521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:25.065932 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:25.065667 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:21:25.065932 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:25.065731 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls podName:21622b2f-0c96-4f74-bc80-832288568931 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:33.065711915 +0000 UTC m=+50.645805805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls") pod "dns-default-5b4ll" (UID: "21622b2f-0c96-4f74-bc80-832288568931") : secret "dns-default-metrics-tls" not found Jul 22 16:21:26.484386 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.484346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:26.542509 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.542481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:26.602363 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.602331 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:26.719700 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.719672 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf"] Jul 22 16:21:26.806972 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.806938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf"] Jul 22 16:21:26.807142 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.807050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:26.811344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.811314 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 16:21:26.811344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.811314 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tgn9s\"" Jul 22 16:21:26.811563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.811354 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 16:21:26.811563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.811376 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 16:21:26.811563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.811315 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 16:21:26.818566 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.818549 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-p9kbk"] Jul 22 16:21:26.843356 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.843323 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54fcd77c68-6z9c2"] Jul 22 16:21:26.843509 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.843463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:26.846902 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.846871 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 16:21:26.847147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.847107 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 16:21:26.847428 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.847390 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 16:21:26.847662 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.847643 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4w2jb\"" Jul 22 16:21:26.848317 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.848257 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 16:21:26.861449 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.861420 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-p9kbk"] Jul 22 16:21:26.861581 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.861535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54fcd77c68-6z9c2"] Jul 22 16:21:26.861581 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.861532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:26.863883 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.863849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 16:21:26.864050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.863898 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bbhl8\"" Jul 22 16:21:26.864050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.863900 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 16:21:26.864050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.864013 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 16:21:26.864220 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.864080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 16:21:26.864220 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.864193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 16:21:26.864351 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.864243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 16:21:26.864351 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.864279 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 16:21:26.924924 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.924893 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76"] Jul 22 16:21:26.939612 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.939589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:26.941993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.941970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:21:26.942136 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.942102 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 16:21:26.942202 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.942139 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 16:21:26.942455 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.942439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-njjbh\"" Jul 22 16:21:26.942672 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.942649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76"] Jul 22 16:21:26.962575 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.962541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:26.979488 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-default-certificate\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:26.979623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78842c1-409e-4a48-a689-901e7f172152-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:26.979623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvct\" (UniqueName: \"kubernetes.io/projected/64f5290c-94c5-4127-816e-9163bb7034a1-kube-api-access-jlvct\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:26.979623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-stats-auth\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:26.979623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64f5290c-94c5-4127-816e-9163bb7034a1-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:26.979773 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:26.979773 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:26.979773 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8d5\" (UniqueName: \"kubernetes.io/projected/29ad3989-b8cf-44f8-99fa-05eaba47aa39-kube-api-access-zr8d5\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:26.979773 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e78842c1-409e-4a48-a689-901e7f172152-serving-cert\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:26.979773 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e78842c1-409e-4a48-a689-901e7f172152-snapshots\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:26.979941 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ftbm\" (UniqueName: \"kubernetes.io/projected/e78842c1-409e-4a48-a689-901e7f172152-kube-api-access-6ftbm\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:26.979941 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e78842c1-409e-4a48-a689-901e7f172152-tmp\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:26.979941 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:26.979941 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:26.979919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78842c1-409e-4a48-a689-901e7f172152-service-ca-bundle\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.047882 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.047848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:27.080905 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.080836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ftbm\" (UniqueName: \"kubernetes.io/projected/e78842c1-409e-4a48-a689-901e7f172152-kube-api-access-6ftbm\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.080905 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.080871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e78842c1-409e-4a48-a689-901e7f172152-tmp\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.081101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.080947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.081101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.080999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78842c1-409e-4a48-a689-901e7f172152-service-ca-bundle\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.081101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-default-certificate\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.081101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78842c1-409e-4a48-a689-901e7f172152-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.081291 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.081133 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:21:27.081291 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.081202 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:27.581177464 +0000 UTC m=+45.161271355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : secret "router-metrics-certs-default" not found Jul 22 16:21:27.081291 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvct\" (UniqueName: \"kubernetes.io/projected/64f5290c-94c5-4127-816e-9163bb7034a1-kube-api-access-jlvct\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:27.081498 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4b5\" (UniqueName: \"kubernetes.io/projected/0003af95-52ac-45e7-b8b0-0d2e869c185f-kube-api-access-5t4b5\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:27.081498 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e78842c1-409e-4a48-a689-901e7f172152-tmp\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.081498 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-stats-auth\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.081498 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64f5290c-94c5-4127-816e-9163bb7034a1-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:27.081675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:27.081675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.081675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:27.081675 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.081652 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:27.081864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8d5\" (UniqueName: \"kubernetes.io/projected/29ad3989-b8cf-44f8-99fa-05eaba47aa39-kube-api-access-zr8d5\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.081864 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.081725 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls podName:64f5290c-94c5-4127-816e-9163bb7034a1 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:27.581703875 +0000 UTC m=+45.161797765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lj4xf" (UID: "64f5290c-94c5-4127-816e-9163bb7034a1") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:27.081864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e78842c1-409e-4a48-a689-901e7f172152-serving-cert\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.081864 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.081813 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:27.581792391 +0000 UTC m=+45.161886267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : configmap references non-existent config key: service-ca.crt Jul 22 16:21:27.082074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e78842c1-409e-4a48-a689-901e7f172152-snapshots\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.082074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.081909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78842c1-409e-4a48-a689-901e7f172152-service-ca-bundle\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.082502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.082477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e78842c1-409e-4a48-a689-901e7f172152-snapshots\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.082845 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.082668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78842c1-409e-4a48-a689-901e7f172152-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.083163 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.083132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/64f5290c-94c5-4127-816e-9163bb7034a1-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:27.085389 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.085361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e78842c1-409e-4a48-a689-901e7f172152-serving-cert\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.085490 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.085429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-default-certificate\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.085490 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.085445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-stats-auth\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.091088 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.091065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8d5\" (UniqueName: \"kubernetes.io/projected/29ad3989-b8cf-44f8-99fa-05eaba47aa39-kube-api-access-zr8d5\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.091088 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.091075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvct\" (UniqueName: \"kubernetes.io/projected/64f5290c-94c5-4127-816e-9163bb7034a1-kube-api-access-jlvct\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:27.091360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.091338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ftbm\" (UniqueName: \"kubernetes.io/projected/e78842c1-409e-4a48-a689-901e7f172152-kube-api-access-6ftbm\") pod \"insights-operator-555d9d7885-p9kbk\" (UID: \"e78842c1-409e-4a48-a689-901e7f172152\") " pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.154851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.154824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" Jul 22 16:21:27.182237 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.182208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4b5\" (UniqueName: \"kubernetes.io/projected/0003af95-52ac-45e7-b8b0-0d2e869c185f-kube-api-access-5t4b5\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:27.182379 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.182265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:27.182467 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.182455 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:21:27.182533 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.182520 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls podName:0003af95-52ac-45e7-b8b0-0d2e869c185f nodeName:}" failed. No retries permitted until 2026-07-22 16:21:27.682504552 +0000 UTC m=+45.262598430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wtp76" (UID: "0003af95-52ac-45e7-b8b0-0d2e869c185f") : secret "samples-operator-tls" not found Jul 22 16:21:27.193662 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.193637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4b5\" (UniqueName: \"kubernetes.io/projected/0003af95-52ac-45e7-b8b0-0d2e869c185f-kube-api-access-5t4b5\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:27.302263 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.302214 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-p9kbk"] Jul 22 16:21:27.305716 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:27.305692 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78842c1_409e_4a48_a689_901e7f172152.slice/crio-cefcf7add3b8691b4f6ae6a50444d2fde2b5baa644cebb17448f0b8ab93f141f WatchSource:0}: Error finding container cefcf7add3b8691b4f6ae6a50444d2fde2b5baa644cebb17448f0b8ab93f141f: Status 404 returned error can't find the container with id cefcf7add3b8691b4f6ae6a50444d2fde2b5baa644cebb17448f0b8ab93f141f Jul 22 16:21:27.584900 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.584867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.584907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.584981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.585011 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.585073 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls podName:64f5290c-94c5-4127-816e-9163bb7034a1 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:28.585057617 +0000 UTC m=+46.165151493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lj4xf" (UID: "64f5290c-94c5-4127-816e-9163bb7034a1") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.585096 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.585099 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:28.585084151 +0000 UTC m=+46.165178037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : configmap references non-existent config key: service-ca.crt Jul 22 16:21:27.585290 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.585140 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:28.585131131 +0000 UTC m=+46.165225006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : secret "router-metrics-certs-default" not found Jul 22 16:21:27.685966 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:27.685931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:27.686117 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.686081 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:21:27.686160 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:27.686146 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls podName:0003af95-52ac-45e7-b8b0-0d2e869c185f nodeName:}" failed. No retries permitted until 2026-07-22 16:21:28.686128833 +0000 UTC m=+46.266222723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wtp76" (UID: "0003af95-52ac-45e7-b8b0-0d2e869c185f") : secret "samples-operator-tls" not found Jul 22 16:21:28.158539 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:28.158496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" event={"ID":"e78842c1-409e-4a48-a689-901e7f172152","Type":"ContainerStarted","Data":"cefcf7add3b8691b4f6ae6a50444d2fde2b5baa644cebb17448f0b8ab93f141f"} Jul 22 16:21:28.594009 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:28.593964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:28.594069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.594169 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.594171 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.594231 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls podName:64f5290c-94c5-4127-816e-9163bb7034a1 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:30.594212369 +0000 UTC m=+48.174306250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lj4xf" (UID: "64f5290c-94c5-4127-816e-9163bb7034a1") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:28.594250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.594359 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:30.594348575 +0000 UTC m=+48.174442449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : secret "router-metrics-certs-default" not found Jul 22 16:21:28.594524 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.594377 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:30.594368602 +0000 UTC m=+48.174462485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : configmap references non-existent config key: service-ca.crt Jul 22 16:21:28.695554 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:28.695518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:28.695714 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.695654 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:21:28.695757 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:28.695733 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls podName:0003af95-52ac-45e7-b8b0-0d2e869c185f nodeName:}" failed. No retries permitted until 2026-07-22 16:21:30.695715787 +0000 UTC m=+48.275809665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wtp76" (UID: "0003af95-52ac-45e7-b8b0-0d2e869c185f") : secret "samples-operator-tls" not found Jul 22 16:21:30.165456 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:30.165351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" event={"ID":"e78842c1-409e-4a48-a689-901e7f172152","Type":"ContainerStarted","Data":"e72a26245d15d2e329e14df7d0d61bbbf0ad67854ebfb728c13c237615f23e96"} Jul 22 16:21:30.183336 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:30.183273 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" podStartSLOduration=1.704466562 podStartE2EDuration="4.183256216s" totalImagesPullingTime="2.478789654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:30.18318756 +0000 UTC m=+47.763281494" watchObservedRunningTime="2026-07-22 16:21:30.183256216 +0000 UTC m=+47.763350173" Jul 22 16:21:30.612611 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:30.612568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:30.612820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:30.612631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:30.612820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:30.612698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:30.612820 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.612722 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:30.612820 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.612769 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls podName:64f5290c-94c5-4127-816e-9163bb7034a1 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:34.612756064 +0000 UTC m=+52.192849939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lj4xf" (UID: "64f5290c-94c5-4127-816e-9163bb7034a1") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:30.613000 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.612827 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:21:30.613000 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.612865 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:34.612849747 +0000 UTC m=+52.192943623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : configmap references non-existent config key: service-ca.crt Jul 22 16:21:30.613000 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.612888 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:34.612877717 +0000 UTC m=+52.192971595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : secret "router-metrics-certs-default" not found Jul 22 16:21:30.713748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:30.713710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:30.713903 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.713824 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:21:30.713903 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:30.713875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls podName:0003af95-52ac-45e7-b8b0-0d2e869c185f nodeName:}" failed. No retries permitted until 2026-07-22 16:21:34.71386232 +0000 UTC m=+52.293956195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wtp76" (UID: "0003af95-52ac-45e7-b8b0-0d2e869c185f") : secret "samples-operator-tls" not found Jul 22 16:21:31.236979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.236949 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v89rw"] Jul 22 16:21:31.259732 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.259702 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v89rw"] Jul 22 16:21:31.259889 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.259843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.261966 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.261945 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 16:21:31.262091 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.261948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 16:21:31.262091 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.261992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-km9pf\"" Jul 22 16:21:31.419893 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.419854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vtg\" (UniqueName: \"kubernetes.io/projected/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-api-access-z8vtg\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.420065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.419981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bff7c473-0eea-4e23-85f8-0e7162fe924c-data-volume\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.420065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.420033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.420065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.420057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.420290 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.420184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bff7c473-0eea-4e23-85f8-0e7162fe924c-crio-socket\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521372 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bff7c473-0eea-4e23-85f8-0e7162fe924c-crio-socket\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vtg\" (UniqueName: \"kubernetes.io/projected/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-api-access-z8vtg\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bff7c473-0eea-4e23-85f8-0e7162fe924c-data-volume\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bff7c473-0eea-4e23-85f8-0e7162fe924c-crio-socket\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.521734 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:31.521596 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:21:31.521734 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:31.521597 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:21:31.521734 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:31.521671 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:32.021650505 +0000 UTC m=+49.601744396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : configmap "kube-rbac-proxy" not found Jul 22 16:21:31.521734 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:31.521729 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:32.021712586 +0000 UTC m=+49.601806475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : secret "insights-runtime-extractor-tls" not found Jul 22 16:21:31.521936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.521847 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bff7c473-0eea-4e23-85f8-0e7162fe924c-data-volume\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:31.531296 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:31.531266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vtg\" (UniqueName: \"kubernetes.io/projected/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-api-access-z8vtg\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:32.025275 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:32.025242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:32.025275 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:32.025278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:32.025484 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:32.025366 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:21:32.025484 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:32.025450 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:33.025432031 +0000 UTC m=+50.605525922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : configmap "kube-rbac-proxy" not found Jul 22 16:21:32.025554 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:32.025491 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:21:32.025554 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:32.025543 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:33.025531837 +0000 UTC m=+50.605625712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : secret "insights-runtime-extractor-tls" not found Jul 22 16:21:33.018168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:33.018136 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmmv2_8e375cc0-38fb-4257-8a70-bb24353d6f64/dns-node-resolver/0.log" Jul 22 16:21:33.033975 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:33.033948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:33.034120 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:33.034081 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:21:33.034181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:33.034132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:33.034181 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:33.034161 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:35.034145035 +0000 UTC m=+52.614238910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : secret "insights-runtime-extractor-tls" not found Jul 22 16:21:33.034264 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:33.034219 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:21:33.034306 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:33.034270 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:35.034256688 +0000 UTC m=+52.614350563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : configmap "kube-rbac-proxy" not found Jul 22 16:21:33.135083 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:33.135044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:33.135244 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:33.135194 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:21:33.135283 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:33.135259 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls podName:21622b2f-0c96-4f74-bc80-832288568931 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:49.13524293 +0000 UTC m=+66.715336806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls") pod "dns-default-5b4ll" (UID: "21622b2f-0c96-4f74-bc80-832288568931") : secret "dns-default-metrics-tls" not found Jul 22 16:21:33.620699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:33.620670 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqpjx_826c741a-501c-4c72-a2e9-a793749a666f/node-ca/0.log" Jul 22 16:21:34.218432 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.218380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7x4pm_e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664/serve-healthcheck-canary/0.log" Jul 22 16:21:34.647493 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.647460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:34.647661 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.647625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:34.647661 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.647634 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:42.647614991 +0000 UTC m=+60.227708865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : configmap references non-existent config key: service-ca.crt Jul 22 16:21:34.647748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.647731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:34.647782 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.647760 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:34.647814 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.647807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls podName:64f5290c-94c5-4127-816e-9163bb7034a1 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:42.647793657 +0000 UTC m=+60.227887535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lj4xf" (UID: "64f5290c-94c5-4127-816e-9163bb7034a1") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:34.647865 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.647852 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:21:34.647908 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.647898 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:42.647886726 +0000 UTC m=+60.227980605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : secret "router-metrics-certs-default" not found Jul 22 16:21:34.748106 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.748066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:34.748259 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.748218 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:21:34.748301 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:34.748285 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls podName:0003af95-52ac-45e7-b8b0-0d2e869c185f nodeName:}" failed. No retries permitted until 2026-07-22 16:21:42.748267298 +0000 UTC m=+60.328361190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wtp76" (UID: "0003af95-52ac-45e7-b8b0-0d2e869c185f") : secret "samples-operator-tls" not found Jul 22 16:21:34.817766 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.817733 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2"] Jul 22 16:21:34.833431 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.833383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2"] Jul 22 16:21:34.833543 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.833473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:34.835664 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.835641 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 16:21:34.835846 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.835750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 16:21:34.835937 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.835876 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 16:21:34.836623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.836606 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-f4h6p\"" Jul 22 16:21:34.836708 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.836630 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:21:34.949419 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.949326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcrb\" (UniqueName: \"kubernetes.io/projected/4c3cd233-f901-441f-ba35-df1a4a9e9c09-kube-api-access-7wcrb\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:34.949419 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.949419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c3cd233-f901-441f-ba35-df1a4a9e9c09-config\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:34.949592 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:34.949496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3cd233-f901-441f-ba35-df1a4a9e9c09-serving-cert\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.027901 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.027868 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6495bccd87-68tbm"] Jul 22 16:21:35.042877 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.042849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.043565 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.043545 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6495bccd87-68tbm"] Jul 22 16:21:35.045192 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.045175 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 16:21:35.045288 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.045247 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-88qfg\"" Jul 22 16:21:35.045348 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.045299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 16:21:35.045781 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.045760 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.050871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcrb\" (UniqueName: \"kubernetes.io/projected/4c3cd233-f901-441f-ba35-df1a4a9e9c09-kube-api-access-7wcrb\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.051054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.051091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.051227 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.051228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c3cd233-f901-441f-ba35-df1a4a9e9c09-config\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.051326 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:39.0513034 +0000 UTC m=+56.631397288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : configmap "kube-rbac-proxy" not found Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.051374 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.051425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3cd233-f901-441f-ba35-df1a4a9e9c09-serving-cert\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.051133 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.051455 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:39.051435201 +0000 UTC m=+56.631529090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : secret "insights-runtime-extractor-tls" not found Jul 22 16:21:35.052481 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.052033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c3cd233-f901-441f-ba35-df1a4a9e9c09-config\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.053726 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.053699 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 16:21:35.054615 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.054594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3cd233-f901-441f-ba35-df1a4a9e9c09-serving-cert\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.059083 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.059065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcrb\" (UniqueName: \"kubernetes.io/projected/4c3cd233-f901-441f-ba35-df1a4a9e9c09-kube-api-access-7wcrb\") pod \"service-ca-operator-5bcbc94c78-szkq2\" (UID: \"4c3cd233-f901-441f-ba35-df1a4a9e9c09\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.142473 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.142438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" Jul 22 16:21:35.152731 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-certificates\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152800 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-trusted-ca\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152800 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c3656eb-3a29-4ed5-9748-676c2922e7d8-ca-trust-extracted\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152896 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-installation-pull-secrets\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152896 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r255j\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-kube-api-access-r255j\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152953 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-image-registry-private-configuration\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.152999 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.152977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-bound-sa-token\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.253728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-certificates\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.253775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-trusted-ca\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.253831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.253952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c3656eb-3a29-4ed5-9748-676c2922e7d8-ca-trust-extracted\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.253976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-installation-pull-secrets\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.254010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r255j\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-kube-api-access-r255j\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.254080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-image-registry-private-configuration\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.254114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-bound-sa-token\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.254514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-certificates\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.254702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c3656eb-3a29-4ed5-9748-676c2922e7d8-ca-trust-extracted\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.254797 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.254735 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:21:35.254797 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.254765 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6495bccd87-68tbm: secret "image-registry-tls" not found Jul 22 16:21:35.255063 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.254861 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls podName:0c3656eb-3a29-4ed5-9748-676c2922e7d8 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:35.754837362 +0000 UTC m=+53.334931253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls") pod "image-registry-6495bccd87-68tbm" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8") : secret "image-registry-tls" not found Jul 22 16:21:35.255974 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.255951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-trusted-ca\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.257446 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.257380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-installation-pull-secrets\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.260187 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.260162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-image-registry-private-configuration\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.263281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.263253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-bound-sa-token\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.263661 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.263645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r255j\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-kube-api-access-r255j\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.299959 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.299921 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2"] Jul 22 16:21:35.302758 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:35.302733 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3cd233_f901_441f_ba35_df1a4a9e9c09.slice/crio-75395f95be3d5482d3c6a7efb848a87875d66ed67d99d77b5c3c466b2705d705 WatchSource:0}: Error finding container 75395f95be3d5482d3c6a7efb848a87875d66ed67d99d77b5c3c466b2705d705: Status 404 returned error can't find the container with id 75395f95be3d5482d3c6a7efb848a87875d66ed67d99d77b5c3c466b2705d705 Jul 22 16:21:35.758659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:35.758622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:35.758882 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.758763 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:21:35.758882 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.758784 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6495bccd87-68tbm: secret "image-registry-tls" not found Jul 22 16:21:35.758882 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:35.758838 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls podName:0c3656eb-3a29-4ed5-9748-676c2922e7d8 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:36.75882239 +0000 UTC m=+54.338916268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls") pod "image-registry-6495bccd87-68tbm" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8") : secret "image-registry-tls" not found Jul 22 16:21:36.181540 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:36.181445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" event={"ID":"4c3cd233-f901-441f-ba35-df1a4a9e9c09","Type":"ContainerStarted","Data":"75395f95be3d5482d3c6a7efb848a87875d66ed67d99d77b5c3c466b2705d705"} Jul 22 16:21:36.766504 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:36.766468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:36.766920 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:36.766653 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:21:36.766920 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:36.766674 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6495bccd87-68tbm: secret "image-registry-tls" not found Jul 22 16:21:36.766920 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:36.766756 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls podName:0c3656eb-3a29-4ed5-9748-676c2922e7d8 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:38.76673302 +0000 UTC m=+56.346826909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls") pod "image-registry-6495bccd87-68tbm" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8") : secret "image-registry-tls" not found Jul 22 16:21:38.188388 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:38.188343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" event={"ID":"4c3cd233-f901-441f-ba35-df1a4a9e9c09","Type":"ContainerStarted","Data":"47cb7a69d63ffd23156453d22be67bb9b5df85ca19577c215bab54207dec0da3"} Jul 22 16:21:38.204894 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:38.204842 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" podStartSLOduration=2.167158913 podStartE2EDuration="4.204825637s" totalImagesPullingTime="2.037666724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:38.204324217 +0000 UTC m=+55.784418116" watchObservedRunningTime="2026-07-22 16:21:38.204825637 +0000 UTC m=+55.784919534" Jul 22 16:21:38.786801 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:38.786761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:38.786975 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:38.786902 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:21:38.786975 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:38.786914 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6495bccd87-68tbm: secret "image-registry-tls" not found Jul 22 16:21:38.786975 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:38.786966 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls podName:0c3656eb-3a29-4ed5-9748-676c2922e7d8 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:42.786951028 +0000 UTC m=+60.367044903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls") pod "image-registry-6495bccd87-68tbm" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8") : secret "image-registry-tls" not found Jul 22 16:21:39.089104 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:39.089010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:39.089104 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:39.089059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:39.089275 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:39.089142 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:21:39.089275 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:39.089212 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:21:39.089275 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:39.089219 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:47.089199247 +0000 UTC m=+64.669293139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : configmap "kube-rbac-proxy" not found Jul 22 16:21:39.089275 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:39.089250 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:21:47.089239148 +0000 UTC m=+64.669333026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : secret "insights-runtime-extractor-tls" not found Jul 22 16:21:42.124345 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:42.124320 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cgh" Jul 22 16:21:42.718875 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:42.718835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:42.719056 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.718992 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:42.719056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:42.719006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:42.719179 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.719061 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls podName:64f5290c-94c5-4127-816e-9163bb7034a1 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:58.719042009 +0000 UTC m=+76.299135884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lj4xf" (UID: "64f5290c-94c5-4127-816e-9163bb7034a1") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:21:42.719179 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:42.719095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:42.719179 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.719139 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 16:21:42.719302 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.719188 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:58.719176306 +0000 UTC m=+76.299270182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : secret "router-metrics-certs-default" not found Jul 22 16:21:42.719302 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.719229 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle podName:29ad3989-b8cf-44f8-99fa-05eaba47aa39 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:58.719215501 +0000 UTC m=+76.299309375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle") pod "router-default-54fcd77c68-6z9c2" (UID: "29ad3989-b8cf-44f8-99fa-05eaba47aa39") : configmap references non-existent config key: service-ca.crt Jul 22 16:21:42.820140 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:42.820111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:42.820309 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:42.820191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:42.820309 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.820264 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:21:42.820309 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.820282 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6495bccd87-68tbm: secret "image-registry-tls" not found Jul 22 16:21:42.820309 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.820302 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:21:42.820482 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.820330 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls podName:0c3656eb-3a29-4ed5-9748-676c2922e7d8 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:50.820315128 +0000 UTC m=+68.400409006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls") pod "image-registry-6495bccd87-68tbm" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8") : secret "image-registry-tls" not found Jul 22 16:21:42.820482 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:42.820342 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls podName:0003af95-52ac-45e7-b8b0-0d2e869c185f nodeName:}" failed. No retries permitted until 2026-07-22 16:21:58.820336319 +0000 UTC m=+76.400430194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wtp76" (UID: "0003af95-52ac-45e7-b8b0-0d2e869c185f") : secret "samples-operator-tls" not found Jul 22 16:21:47.153419 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.153356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:47.153800 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.153434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:47.153800 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:47.153597 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:21:47.153800 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:47.153664 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls podName:bff7c473-0eea-4e23-85f8-0e7162fe924c nodeName:}" failed. No retries permitted until 2026-07-22 16:22:03.15364707 +0000 UTC m=+80.733740961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v89rw" (UID: "bff7c473-0eea-4e23-85f8-0e7162fe924c") : secret "insights-runtime-extractor-tls" not found Jul 22 16:21:47.153911 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.153850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bff7c473-0eea-4e23-85f8-0e7162fe924c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:21:47.759416 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.759362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:47.759580 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.759479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:21:47.761934 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.761912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:21:47.761977 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.761912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:21:47.769932 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:47.769916 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 16:21:47.769994 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:21:47.769979 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs podName:d79635de-e4ce-4ebc-9921-b7e89ccd30bb nodeName:}" failed. No retries permitted until 2026-07-22 16:22:51.76996081 +0000 UTC m=+129.350054686 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs") pod "network-metrics-daemon-kfn8q" (UID: "d79635de-e4ce-4ebc-9921-b7e89ccd30bb") : secret "metrics-daemon-secret" not found Jul 22 16:21:47.771982 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.771954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/269497b7-5123-4d80-8735-471d64f1e566-original-pull-secret\") pod \"global-pull-secret-syncer-l2gk6\" (UID: \"269497b7-5123-4d80-8735-471d64f1e566\") " pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:47.860688 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.860648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:47.863154 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.863136 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:21:47.872932 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.872909 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:21:47.883974 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.883953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n629f\" (UniqueName: \"kubernetes.io/projected/f4123489-a5d6-4004-8315-36707890bd18-kube-api-access-n629f\") pod \"network-check-target-z6s8g\" (UID: \"f4123489-a5d6-4004-8315-36707890bd18\") " pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:47.983752 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.983713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2gk6" Jul 22 16:21:47.996916 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:47.996895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6qdrm\"" Jul 22 16:21:48.004993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:48.004962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:48.113966 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:48.113936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2gk6"] Jul 22 16:21:48.117019 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:48.116993 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269497b7_5123_4d80_8735_471d64f1e566.slice/crio-82cc6a910f2901f47386a68be3e234c93d8f03e0ec9ae4bff7575b67106807ca WatchSource:0}: Error finding container 82cc6a910f2901f47386a68be3e234c93d8f03e0ec9ae4bff7575b67106807ca: Status 404 returned error can't find the container with id 82cc6a910f2901f47386a68be3e234c93d8f03e0ec9ae4bff7575b67106807ca Jul 22 16:21:48.138802 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:48.138780 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z6s8g"] Jul 22 16:21:48.142051 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:48.142027 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4123489_a5d6_4004_8315_36707890bd18.slice/crio-f525914e96fbe341b574fe43b1f57669a721304bd4b952ec63d091bdbbbb7caa WatchSource:0}: Error finding container f525914e96fbe341b574fe43b1f57669a721304bd4b952ec63d091bdbbbb7caa: Status 404 returned error can't find the container with id f525914e96fbe341b574fe43b1f57669a721304bd4b952ec63d091bdbbbb7caa Jul 22 16:21:48.214083 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:48.214044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z6s8g" event={"ID":"f4123489-a5d6-4004-8315-36707890bd18","Type":"ContainerStarted","Data":"f525914e96fbe341b574fe43b1f57669a721304bd4b952ec63d091bdbbbb7caa"} Jul 22 16:21:48.215080 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:48.215059 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2gk6" event={"ID":"269497b7-5123-4d80-8735-471d64f1e566","Type":"ContainerStarted","Data":"82cc6a910f2901f47386a68be3e234c93d8f03e0ec9ae4bff7575b67106807ca"} Jul 22 16:21:49.173154 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:49.173114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:49.176507 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:49.176483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21622b2f-0c96-4f74-bc80-832288568931-metrics-tls\") pod \"dns-default-5b4ll\" (UID: \"21622b2f-0c96-4f74-bc80-832288568931\") " pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:49.392261 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:49.392227 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4ntcb\"" Jul 22 16:21:49.401255 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:49.401194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:49.572816 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:49.572668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5b4ll"] Jul 22 16:21:49.575904 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:49.575872 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21622b2f_0c96_4f74_bc80_832288568931.slice/crio-ff624672d33bce9de0a66697b6187cbb33d66d4f43c188d5f73085b7708f8115 WatchSource:0}: Error finding container ff624672d33bce9de0a66697b6187cbb33d66d4f43c188d5f73085b7708f8115: Status 404 returned error can't find the container with id ff624672d33bce9de0a66697b6187cbb33d66d4f43c188d5f73085b7708f8115 Jul 22 16:21:50.223033 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:50.222993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5b4ll" event={"ID":"21622b2f-0c96-4f74-bc80-832288568931","Type":"ContainerStarted","Data":"ff624672d33bce9de0a66697b6187cbb33d66d4f43c188d5f73085b7708f8115"} Jul 22 16:21:50.889281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:50.889232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:50.892385 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:50.892331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"image-registry-6495bccd87-68tbm\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:50.957662 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:50.957626 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-88qfg\"" Jul 22 16:21:50.966549 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:50.966520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:53.326116 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:53.326079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6495bccd87-68tbm"] Jul 22 16:21:53.331658 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:53.331632 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3656eb_3a29_4ed5_9748_676c2922e7d8.slice/crio-425dd676b3cf7c9f18ba1abd4ac9b6bbbd94c2d97c0bb8a61c39d7197faa8aa6 WatchSource:0}: Error finding container 425dd676b3cf7c9f18ba1abd4ac9b6bbbd94c2d97c0bb8a61c39d7197faa8aa6: Status 404 returned error can't find the container with id 425dd676b3cf7c9f18ba1abd4ac9b6bbbd94c2d97c0bb8a61c39d7197faa8aa6 Jul 22 16:21:54.241431 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.241372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2gk6" event={"ID":"269497b7-5123-4d80-8735-471d64f1e566","Type":"ContainerStarted","Data":"e2b3f5cdb05646e882a009856b0df4e795aa3c71a53b72fe305150684130b6da"} Jul 22 16:21:54.242810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.242786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" event={"ID":"0c3656eb-3a29-4ed5-9748-676c2922e7d8","Type":"ContainerStarted","Data":"423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727"} Jul 22 16:21:54.242928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.242827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" event={"ID":"0c3656eb-3a29-4ed5-9748-676c2922e7d8","Type":"ContainerStarted","Data":"425dd676b3cf7c9f18ba1abd4ac9b6bbbd94c2d97c0bb8a61c39d7197faa8aa6"} Jul 22 16:21:54.242928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.242892 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:21:54.244153 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.244133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z6s8g" event={"ID":"f4123489-a5d6-4004-8315-36707890bd18","Type":"ContainerStarted","Data":"a40672f67810cb5cbce35e3d08d3732929bbb87a09e1d4b65b5b78caee665613"} Jul 22 16:21:54.244252 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.244181 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:21:54.245788 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.245766 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5b4ll" event={"ID":"21622b2f-0c96-4f74-bc80-832288568931","Type":"ContainerStarted","Data":"09d328e1dfb0b7833c73f00ffe99333c23c1d3270ba6ba3aef75650935e53332"} Jul 22 16:21:54.245862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.245810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5b4ll" event={"ID":"21622b2f-0c96-4f74-bc80-832288568931","Type":"ContainerStarted","Data":"b116bf8b78ed08422863e4d5ab165511e5e0ac13892c83ebb47ebec27010e9f3"} Jul 22 16:21:54.245862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.245851 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5b4ll" Jul 22 16:21:54.257952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.257915 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l2gk6" podStartSLOduration=66.18090158 podStartE2EDuration="1m11.25790154s" totalImagesPullingTime="5.07699996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:54.257245634 +0000 UTC m=+71.837339532" watchObservedRunningTime="2026-07-22 16:21:54.25790154 +0000 UTC m=+71.837995437" Jul 22 16:21:54.273753 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.273710 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z6s8g" podStartSLOduration=66.241323857 podStartE2EDuration="1m11.27369621s" totalImagesPullingTime="5.032372353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:54.272674324 +0000 UTC m=+71.852768237" watchObservedRunningTime="2026-07-22 16:21:54.27369621 +0000 UTC m=+71.853790107" Jul 22 16:21:54.289872 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.289828 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5b4ll" podStartSLOduration=33.691118439 podStartE2EDuration="37.289811222s" totalImagesPullingTime="3.598692783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:54.288759805 +0000 UTC m=+71.868853702" watchObservedRunningTime="2026-07-22 16:21:54.289811222 +0000 UTC m=+71.869905120" Jul 22 16:21:54.308677 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:54.308630 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" podStartSLOduration=19.30861716 podStartE2EDuration="19.30861716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:54.307905252 +0000 UTC m=+71.887999168" watchObservedRunningTime="2026-07-22 16:21:54.30861716 +0000 UTC m=+71.888711057" Jul 22 16:21:58.756525 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.756469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:58.756931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.756611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:58.756931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.756642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:58.757236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.757218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ad3989-b8cf-44f8-99fa-05eaba47aa39-service-ca-bundle\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:58.758949 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.758922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/64f5290c-94c5-4127-816e-9163bb7034a1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lj4xf\" (UID: \"64f5290c-94c5-4127-816e-9163bb7034a1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:58.758949 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.758936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ad3989-b8cf-44f8-99fa-05eaba47aa39-metrics-certs\") pod \"router-default-54fcd77c68-6z9c2\" (UID: \"29ad3989-b8cf-44f8-99fa-05eaba47aa39\") " pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:58.857568 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.857508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:58.859826 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.859791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0003af95-52ac-45e7-b8b0-0d2e869c185f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wtp76\" (UID: \"0003af95-52ac-45e7-b8b0-0d2e869c185f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:58.918158 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.918126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tgn9s\"" Jul 22 16:21:58.927019 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.926995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" Jul 22 16:21:58.973457 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.973426 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bbhl8\"" Jul 22 16:21:58.980496 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:58.980462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:59.051561 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.051537 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-njjbh\"" Jul 22 16:21:59.057676 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.057654 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf"] Jul 22 16:21:59.059504 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.059472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" Jul 22 16:21:59.059838 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:59.059816 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f5290c_94c5_4127_816e_9163bb7034a1.slice/crio-0bdd6404f1b9f69de24415dd6ec09bf25da1453d847c6ed1583a3593f83b2acd WatchSource:0}: Error finding container 0bdd6404f1b9f69de24415dd6ec09bf25da1453d847c6ed1583a3593f83b2acd: Status 404 returned error can't find the container with id 0bdd6404f1b9f69de24415dd6ec09bf25da1453d847c6ed1583a3593f83b2acd Jul 22 16:21:59.118764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.118668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54fcd77c68-6z9c2"] Jul 22 16:21:59.122673 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:21:59.122648 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ad3989_b8cf_44f8_99fa_05eaba47aa39.slice/crio-b609443edce07c895688cf3d1dc5483f31aae754af4878f3b14697626ad72214 WatchSource:0}: Error finding container b609443edce07c895688cf3d1dc5483f31aae754af4878f3b14697626ad72214: Status 404 returned error can't find the container with id b609443edce07c895688cf3d1dc5483f31aae754af4878f3b14697626ad72214 Jul 22 16:21:59.196159 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.196131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76"] Jul 22 16:21:59.263436 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.263411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" event={"ID":"29ad3989-b8cf-44f8-99fa-05eaba47aa39","Type":"ContainerStarted","Data":"1ac8e9a46f41b3be7a73cb4b6a9d30437f2c27dbdd806ae9104fd43ea117e765"} Jul 22 16:21:59.263553 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.263458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" event={"ID":"29ad3989-b8cf-44f8-99fa-05eaba47aa39","Type":"ContainerStarted","Data":"b609443edce07c895688cf3d1dc5483f31aae754af4878f3b14697626ad72214"} Jul 22 16:21:59.264486 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.264465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" event={"ID":"0003af95-52ac-45e7-b8b0-0d2e869c185f","Type":"ContainerStarted","Data":"723e18358d36e3a748933a2ffbb220d141a15efda5e461fd084c95410b963f68"} Jul 22 16:21:59.265482 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.265466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" event={"ID":"64f5290c-94c5-4127-816e-9163bb7034a1","Type":"ContainerStarted","Data":"0bdd6404f1b9f69de24415dd6ec09bf25da1453d847c6ed1583a3593f83b2acd"} Jul 22 16:21:59.282958 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.282885 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" podStartSLOduration=33.282865158 podStartE2EDuration="33.282865158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:59.282665584 +0000 UTC m=+76.862759486" watchObservedRunningTime="2026-07-22 16:21:59.282865158 +0000 UTC m=+76.862959057" Jul 22 16:21:59.981710 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.981673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:21:59.984665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:21:59.984640 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:22:00.270440 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:00.270364 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:22:00.271768 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:00.271742 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54fcd77c68-6z9c2" Jul 22 16:22:02.231528 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.231496 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6495bccd87-68tbm"] Jul 22 16:22:02.261490 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.261461 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c54c58fd-zhd9r"] Jul 22 16:22:02.265329 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.265307 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.276929 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.276902 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c54c58fd-zhd9r"] Jul 22 16:22:02.281630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.281602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" event={"ID":"0003af95-52ac-45e7-b8b0-0d2e869c185f","Type":"ContainerStarted","Data":"c7d1f8b2929288b469f527280524d70845740b61924a106d9dce434a1df708f7"} Jul 22 16:22:02.281732 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.281657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" event={"ID":"0003af95-52ac-45e7-b8b0-0d2e869c185f","Type":"ContainerStarted","Data":"0be703c1623f34c0d22516de016771bc26f9a2932b3a185886f276e5b86f2f30"} Jul 22 16:22:02.283228 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.283203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" event={"ID":"64f5290c-94c5-4127-816e-9163bb7034a1","Type":"ContainerStarted","Data":"dc15b6a4c994825bebe2306ebd56c67a8ebddfbc2c40338a0a3a9a8c7b0a2726"} Jul 22 16:22:02.302313 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.302288 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-722k6"] Jul 22 16:22:02.306375 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.306357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:02.308537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.308518 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 16:22:02.308537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.308531 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4mzb9\"" Jul 22 16:22:02.308694 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.308562 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 16:22:02.318743 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.318719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-722k6"] Jul 22 16:22:02.346156 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.346107 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lj4xf" podStartSLOduration=34.192469814 podStartE2EDuration="36.346091485s" totalImagesPullingTime="2.153621671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:02.34494926 +0000 UTC m=+79.925043176" watchObservedRunningTime="2026-07-22 16:22:02.346091485 +0000 UTC m=+79.926185382" Jul 22 16:22:02.387119 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwts\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-kube-api-access-gzwts\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387292 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-bound-sa-token\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387292 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ab3b9f1-342f-4985-a599-051e115a06a0-trusted-ca\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387475 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ab3b9f1-342f-4985-a599-051e115a06a0-ca-trust-extracted\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469d6\" (UniqueName: \"kubernetes.io/projected/2e056b28-ef60-4e85-8c47-f17c5e01ed35-kube-api-access-469d6\") pod \"downloads-5bc67dcd46-722k6\" (UID: \"2e056b28-ef60-4e85-8c47-f17c5e01ed35\") " pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:02.387613 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ab3b9f1-342f-4985-a599-051e115a06a0-registry-certificates\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387782 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-registry-tls\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387782 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ab3b9f1-342f-4985-a599-051e115a06a0-image-registry-private-configuration\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.387782 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.387768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ab3b9f1-342f-4985-a599-051e115a06a0-installation-pull-secrets\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.406720 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.406671 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wtp76" podStartSLOduration=34.435839117 podStartE2EDuration="36.40665607s" totalImagesPullingTime="1.970816953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:02.4058434 +0000 UTC m=+79.985937298" watchObservedRunningTime="2026-07-22 16:22:02.40665607 +0000 UTC m=+79.986749967" Jul 22 16:22:02.489129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ab3b9f1-342f-4985-a599-051e115a06a0-registry-certificates\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489315 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-registry-tls\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489315 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ab3b9f1-342f-4985-a599-051e115a06a0-image-registry-private-configuration\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489315 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ab3b9f1-342f-4985-a599-051e115a06a0-installation-pull-secrets\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489315 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwts\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-kube-api-access-gzwts\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489550 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-bound-sa-token\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489550 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ab3b9f1-342f-4985-a599-051e115a06a0-trusted-ca\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489653 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ab3b9f1-342f-4985-a599-051e115a06a0-ca-trust-extracted\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.489653 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469d6\" (UniqueName: \"kubernetes.io/projected/2e056b28-ef60-4e85-8c47-f17c5e01ed35-kube-api-access-469d6\") pod \"downloads-5bc67dcd46-722k6\" (UID: \"2e056b28-ef60-4e85-8c47-f17c5e01ed35\") " pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:02.489947 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.489919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ab3b9f1-342f-4985-a599-051e115a06a0-ca-trust-extracted\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.490120 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.490096 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ab3b9f1-342f-4985-a599-051e115a06a0-registry-certificates\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.490377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.490362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ab3b9f1-342f-4985-a599-051e115a06a0-trusted-ca\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.491934 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.491905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ab3b9f1-342f-4985-a599-051e115a06a0-image-registry-private-configuration\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.492132 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.492109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-registry-tls\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.492225 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.492112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ab3b9f1-342f-4985-a599-051e115a06a0-installation-pull-secrets\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.499877 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.499851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-469d6\" (UniqueName: \"kubernetes.io/projected/2e056b28-ef60-4e85-8c47-f17c5e01ed35-kube-api-access-469d6\") pod \"downloads-5bc67dcd46-722k6\" (UID: \"2e056b28-ef60-4e85-8c47-f17c5e01ed35\") " pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:02.499992 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.499926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-bound-sa-token\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.499992 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.499970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwts\" (UniqueName: \"kubernetes.io/projected/0ab3b9f1-342f-4985-a599-051e115a06a0-kube-api-access-gzwts\") pod \"image-registry-69c54c58fd-zhd9r\" (UID: \"0ab3b9f1-342f-4985-a599-051e115a06a0\") " pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.574349 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.574308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:02.615370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.615331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:02.712744 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.712714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c54c58fd-zhd9r"] Jul 22 16:22:02.715183 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:02.715154 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab3b9f1_342f_4985_a599_051e115a06a0.slice/crio-a4670fccb9ab6913dc7cbee9b5c58bfb623c929c8ecc21f9530ebb7ccbc9bbb0 WatchSource:0}: Error finding container a4670fccb9ab6913dc7cbee9b5c58bfb623c929c8ecc21f9530ebb7ccbc9bbb0: Status 404 returned error can't find the container with id a4670fccb9ab6913dc7cbee9b5c58bfb623c929c8ecc21f9530ebb7ccbc9bbb0 Jul 22 16:22:02.771262 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:02.771236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-722k6"] Jul 22 16:22:02.774328 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:02.774287 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e056b28_ef60_4e85_8c47_f17c5e01ed35.slice/crio-435918deeeb3983ba75eafd783805e3bb375c2bd844a590f55c0df8e4ad01878 WatchSource:0}: Error finding container 435918deeeb3983ba75eafd783805e3bb375c2bd844a590f55c0df8e4ad01878: Status 404 returned error can't find the container with id 435918deeeb3983ba75eafd783805e3bb375c2bd844a590f55c0df8e4ad01878 Jul 22 16:22:03.198437 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.198312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:22:03.200675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.200648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bff7c473-0eea-4e23-85f8-0e7162fe924c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v89rw\" (UID: \"bff7c473-0eea-4e23-85f8-0e7162fe924c\") " pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:22:03.287690 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.287653 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-722k6" event={"ID":"2e056b28-ef60-4e85-8c47-f17c5e01ed35","Type":"ContainerStarted","Data":"435918deeeb3983ba75eafd783805e3bb375c2bd844a590f55c0df8e4ad01878"} Jul 22 16:22:03.288991 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.288950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" event={"ID":"0ab3b9f1-342f-4985-a599-051e115a06a0","Type":"ContainerStarted","Data":"8196846d9d1959653697948c00fe7ec1b8b416748ee2fb6d48eb8fab77339205"} Jul 22 16:22:03.289109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.288999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" event={"ID":"0ab3b9f1-342f-4985-a599-051e115a06a0","Type":"ContainerStarted","Data":"a4670fccb9ab6913dc7cbee9b5c58bfb623c929c8ecc21f9530ebb7ccbc9bbb0"} Jul 22 16:22:03.310155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.310067 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" podStartSLOduration=1.310046008 podStartE2EDuration="1.310046008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:03.308945453 +0000 UTC m=+80.889039351" watchObservedRunningTime="2026-07-22 16:22:03.310046008 +0000 UTC m=+80.890139909" Jul 22 16:22:03.371195 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.371159 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-km9pf\"" Jul 22 16:22:03.380018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.379984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v89rw" Jul 22 16:22:03.520150 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:03.520117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v89rw"] Jul 22 16:22:03.523551 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:03.523514 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff7c473_0eea_4e23_85f8_0e7162fe924c.slice/crio-34e0dd5b1c4821cda87388cdb29a080df9bb6353d0f2eb0a6e013ce57f617d00 WatchSource:0}: Error finding container 34e0dd5b1c4821cda87388cdb29a080df9bb6353d0f2eb0a6e013ce57f617d00: Status 404 returned error can't find the container with id 34e0dd5b1c4821cda87388cdb29a080df9bb6353d0f2eb0a6e013ce57f617d00 Jul 22 16:22:04.251481 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:04.251448 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5b4ll" Jul 22 16:22:04.298453 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:04.298378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v89rw" event={"ID":"bff7c473-0eea-4e23-85f8-0e7162fe924c","Type":"ContainerStarted","Data":"34e0dd5b1c4821cda87388cdb29a080df9bb6353d0f2eb0a6e013ce57f617d00"} Jul 22 16:22:04.298921 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:04.298591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:10.325878 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:10.325835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v89rw" event={"ID":"bff7c473-0eea-4e23-85f8-0e7162fe924c","Type":"ContainerStarted","Data":"70cd032169f1758e677732f53744d459da6983dfe700e566c501b51fbd05a344"} Jul 22 16:22:12.238884 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:12.238837 2580 patch_prober.go:28] interesting pod/image-registry-6495bccd87-68tbm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 16:22:12.239338 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:12.238900 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" podUID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:22:16.123557 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.123516 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n"] Jul 22 16:22:16.128301 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.128276 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8x8ks"] Jul 22 16:22:16.128456 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.128424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.130704 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.130561 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 16:22:16.130704 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.130575 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 16:22:16.130704 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.130702 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v6w2b\"" Jul 22 16:22:16.131433 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.131413 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 16:22:16.132573 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.132554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.134792 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.134658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 16:22:16.134925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.134855 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 16:22:16.134925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.134877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 16:22:16.135437 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.135250 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjncw\"" Jul 22 16:22:16.139253 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.139234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n"] Jul 22 16:22:16.313099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-textfile\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-accelerators-collector-config\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-wtmp\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njhl\" (UniqueName: \"kubernetes.io/projected/48b032e6-40fc-418e-9041-1e0928a33eec-kube-api-access-2njhl\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.313332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48b032e6-40fc-418e-9041-1e0928a33eec-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.313332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfz4q\" (UniqueName: \"kubernetes.io/projected/c6162922-fec2-43bf-a829-85fdf6b54207-kube-api-access-qfz4q\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.313630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-tls\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.313630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6162922-fec2-43bf-a829-85fdf6b54207-metrics-client-ca\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-root\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.313630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.313538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-sys\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-textfile\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-accelerators-collector-config\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-wtmp\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njhl\" (UniqueName: \"kubernetes.io/projected/48b032e6-40fc-418e-9041-1e0928a33eec-kube-api-access-2njhl\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48b032e6-40fc-418e-9041-1e0928a33eec-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfz4q\" (UniqueName: \"kubernetes.io/projected/c6162922-fec2-43bf-a829-85fdf6b54207-kube-api-access-qfz4q\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-wtmp\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:16.414631 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 22 16:22:16.414699 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:16.414690 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-tls podName:48b032e6-40fc-418e-9041-1e0928a33eec nodeName:}" failed. No retries permitted until 2026-07-22 16:22:16.914671357 +0000 UTC m=+94.494765237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-5t46n" (UID: "48b032e6-40fc-418e-9041-1e0928a33eec") : secret "openshift-state-metrics-tls" not found Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-tls\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6162922-fec2-43bf-a829-85fdf6b54207-metrics-client-ca\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:16.414861 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:16.414905 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-tls podName:c6162922-fec2-43bf-a829-85fdf6b54207 nodeName:}" failed. No retries permitted until 2026-07-22 16:22:16.914890862 +0000 UTC m=+94.494984744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-tls") pod "node-exporter-8x8ks" (UID: "c6162922-fec2-43bf-a829-85fdf6b54207") : secret "node-exporter-tls" not found Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-root\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-sys\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.414997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-textfile\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.415066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-root\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.415066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6162922-fec2-43bf-a829-85fdf6b54207-sys\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415554 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.415188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-accelerators-collector-config\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.415554 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.415307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48b032e6-40fc-418e-9041-1e0928a33eec-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.415554 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.415378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6162922-fec2-43bf-a829-85fdf6b54207-metrics-client-ca\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.416922 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.416899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.417258 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.417241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.423269 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.423252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfz4q\" (UniqueName: \"kubernetes.io/projected/c6162922-fec2-43bf-a829-85fdf6b54207-kube-api-access-qfz4q\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.423526 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.423502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njhl\" (UniqueName: \"kubernetes.io/projected/48b032e6-40fc-418e-9041-1e0928a33eec-kube-api-access-2njhl\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.919276 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.919232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.919489 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.919295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-tls\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:16.921984 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.921935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b032e6-40fc-418e-9041-1e0928a33eec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5t46n\" (UID: \"48b032e6-40fc-418e-9041-1e0928a33eec\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:16.922087 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:16.922044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6162922-fec2-43bf-a829-85fdf6b54207-node-exporter-tls\") pod \"node-exporter-8x8ks\" (UID: \"c6162922-fec2-43bf-a829-85fdf6b54207\") " pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:17.041668 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.041623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" Jul 22 16:22:17.047735 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.047702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8x8ks" Jul 22 16:22:17.210749 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.210660 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:22:17.217823 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.217801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.220465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220229 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 16:22:17.220465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220231 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 16:22:17.220465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220313 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 16:22:17.220465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220333 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 16:22:17.220465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220238 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 16:22:17.220465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220348 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dwzqb\"" Jul 22 16:22:17.220875 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220851 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 16:22:17.220875 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 16:22:17.220986 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.220884 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 16:22:17.221500 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.221481 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 16:22:17.229487 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.229454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:22:17.323266 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-out\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323266 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-web-config\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2gz\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-kube-api-access-zx2gz\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323780 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323780 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323780 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323780 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.323963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.323892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.424582 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.424747 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.424747 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.424747 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.424747 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425052 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425052 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.424790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-out\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425207 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-web-config\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425279 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425279 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425410 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2gz\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-kube-api-access-zx2gz\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425410 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425410 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.425903 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.425875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.426069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.426042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.426193 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.426172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.428060 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.427866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.428060 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.427965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.429138 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.429109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.429244 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.429162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-out\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.429755 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.429732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.429982 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.429961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-web-config\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.430526 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.430494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.430640 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.430580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.430751 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.430727 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.435062 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.435042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2gz\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-kube-api-access-zx2gz\") pod \"alertmanager-main-0\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:17.530952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:17.530911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:22:18.111803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.111766 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-ffbd75db4-lnp9q"] Jul 22 16:22:18.157496 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.157461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ffbd75db4-lnp9q"] Jul 22 16:22:18.157672 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.157601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.159832 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.159807 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 16:22:18.159953 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.159816 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nw9rt\"" Jul 22 16:22:18.160153 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.160021 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 16:22:18.160153 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.160027 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9edlm938c9t3m\"" Jul 22 16:22:18.160371 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.160321 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 16:22:18.160371 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.160321 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 16:22:18.160630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.160613 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 16:22:18.333566 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-tls\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-grpc-tls\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5v2\" (UniqueName: \"kubernetes.io/projected/d0def3e0-1f9b-4983-82aa-2cb760d79c32-kube-api-access-dp5v2\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.334037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.333903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0def3e0-1f9b-4983-82aa-2cb760d79c32-metrics-client-ca\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435231 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-tls\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435231 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435510 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435510 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-grpc-tls\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435510 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5v2\" (UniqueName: \"kubernetes.io/projected/d0def3e0-1f9b-4983-82aa-2cb760d79c32-kube-api-access-dp5v2\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435510 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.435723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.435615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0def3e0-1f9b-4983-82aa-2cb760d79c32-metrics-client-ca\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.436498 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.436444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0def3e0-1f9b-4983-82aa-2cb760d79c32-metrics-client-ca\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.438383 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.438348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.438558 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.438528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.438669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.438559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.438979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.438958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.438979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.438970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-thanos-querier-tls\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.439213 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.439196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0def3e0-1f9b-4983-82aa-2cb760d79c32-secret-grpc-tls\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.443820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.443798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5v2\" (UniqueName: \"kubernetes.io/projected/d0def3e0-1f9b-4983-82aa-2cb760d79c32-kube-api-access-dp5v2\") pod \"thanos-querier-ffbd75db4-lnp9q\" (UID: \"d0def3e0-1f9b-4983-82aa-2cb760d79c32\") " pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:18.469634 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:18.469594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:20.920112 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.919432 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh"] Jul 22 16:22:20.936907 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.936842 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh"] Jul 22 16:22:20.937076 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.937006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:20.940078 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.940039 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xhfg2\"" Jul 22 16:22:20.940475 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.940268 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 16:22:20.966519 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.963429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14523a9d-65dc-4eb8-ab71-be5833fa978d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:20.966519 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:20.963596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4rl\" (UniqueName: \"kubernetes.io/projected/14523a9d-65dc-4eb8-ab71-be5833fa978d-kube-api-access-tn4rl\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:21.011928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.011904 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bcdfb8bb6-gpvgq"] Jul 22 16:22:21.019151 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.019107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:22:21.019544 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.019483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.021527 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:21.021443 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a11e974_b17c_43b7_83fb_c0c5b0027710.slice/crio-2e5dd521bb49185ae5cc20c6dccdb73e40731a8472fdeebc35270a6a6e177a7e WatchSource:0}: Error finding container 2e5dd521bb49185ae5cc20c6dccdb73e40731a8472fdeebc35270a6a6e177a7e: Status 404 returned error can't find the container with id 2e5dd521bb49185ae5cc20c6dccdb73e40731a8472fdeebc35270a6a6e177a7e Jul 22 16:22:21.024218 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.022660 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 16:22:21.024218 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.022721 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 16:22:21.024218 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.022883 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 16:22:21.024218 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.022895 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 16:22:21.024218 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.023029 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bmnj5\"" Jul 22 16:22:21.024218 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.023140 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 16:22:21.030037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.029637 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 16:22:21.030037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.029758 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcdfb8bb6-gpvgq"] Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.064903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-oauth-config\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.064956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4rl\" (UniqueName: \"kubernetes.io/projected/14523a9d-65dc-4eb8-ab71-be5833fa978d-kube-api-access-tn4rl\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.064986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-config\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.065042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14523a9d-65dc-4eb8-ab71-be5833fa978d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.065065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-trusted-ca-bundle\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.065095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-serving-cert\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.065164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-oauth-serving-cert\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:21.065187 2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.065199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbx4c\" (UniqueName: \"kubernetes.io/projected/151e17e2-8a8d-49c3-80aa-44d395d7c47c-kube-api-access-sbx4c\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.065031 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:21.065250 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14523a9d-65dc-4eb8-ab71-be5833fa978d-monitoring-plugin-cert podName:14523a9d-65dc-4eb8-ab71-be5833fa978d nodeName:}" failed. No retries permitted until 2026-07-22 16:22:21.565230685 +0000 UTC m=+99.145324573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/14523a9d-65dc-4eb8-ab71-be5833fa978d-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-f49vh" (UID: "14523a9d-65dc-4eb8-ab71-be5833fa978d") : secret "monitoring-plugin-cert" not found Jul 22 16:22:21.066038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.065374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-service-ca\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.072963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.072941 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4rl\" (UniqueName: \"kubernetes.io/projected/14523a9d-65dc-4eb8-ab71-be5833fa978d-kube-api-access-tn4rl\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:21.076904 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.076882 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ffbd75db4-lnp9q"] Jul 22 16:22:21.080214 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:21.080193 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0def3e0_1f9b_4983_82aa_2cb760d79c32.slice/crio-6129ee0371820ec5ba9e96f9168a47d1ce8822050790ccb4960bac8ea77c3986 WatchSource:0}: Error finding container 6129ee0371820ec5ba9e96f9168a47d1ce8822050790ccb4960bac8ea77c3986: Status 404 returned error can't find the container with id 6129ee0371820ec5ba9e96f9168a47d1ce8822050790ccb4960bac8ea77c3986 Jul 22 16:22:21.099095 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.098997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n"] Jul 22 16:22:21.101467 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:21.101446 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b032e6_40fc_418e_9041_1e0928a33eec.slice/crio-557afc7b20dab5c272037a77cc06dcbda1e33b9dec70770b2906e8f1b1601f8e WatchSource:0}: Error finding container 557afc7b20dab5c272037a77cc06dcbda1e33b9dec70770b2906e8f1b1601f8e: Status 404 returned error can't find the container with id 557afc7b20dab5c272037a77cc06dcbda1e33b9dec70770b2906e8f1b1601f8e Jul 22 16:22:21.165973 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.165951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbx4c\" (UniqueName: \"kubernetes.io/projected/151e17e2-8a8d-49c3-80aa-44d395d7c47c-kube-api-access-sbx4c\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.165991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-service-ca\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.166017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-oauth-config\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.166047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-config\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.166110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-trusted-ca-bundle\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.166139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-serving-cert\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166467 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.166204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-oauth-serving-cert\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.166769 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.166725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-service-ca\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.167157 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.167134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-oauth-serving-cert\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.167157 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.167147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-trusted-ca-bundle\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.167263 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.167193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-config\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.168857 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.168838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-oauth-config\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.168945 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.168871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-serving-cert\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.174727 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.174703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbx4c\" (UniqueName: \"kubernetes.io/projected/151e17e2-8a8d-49c3-80aa-44d395d7c47c-kube-api-access-sbx4c\") pod \"console-6bcdfb8bb6-gpvgq\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.350054 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.349931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:21.371983 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.371910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8x8ks" event={"ID":"c6162922-fec2-43bf-a829-85fdf6b54207","Type":"ContainerStarted","Data":"0eba78c648a1df7e06e723c74a97d9fba2d33eda6dd874628af64363f9ab7a7b"} Jul 22 16:22:21.375099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.374959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" event={"ID":"48b032e6-40fc-418e-9041-1e0928a33eec","Type":"ContainerStarted","Data":"e786e191202e25838a31098ac7f76ff9d1e07eba30d1718e9060ed104ce228b9"} Jul 22 16:22:21.375099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.375020 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" event={"ID":"48b032e6-40fc-418e-9041-1e0928a33eec","Type":"ContainerStarted","Data":"84782751b0c5484c9a9fd53e0cf72185e3039a034a6ba7d58ea1400ca8282491"} Jul 22 16:22:21.375099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.375047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" event={"ID":"48b032e6-40fc-418e-9041-1e0928a33eec","Type":"ContainerStarted","Data":"557afc7b20dab5c272037a77cc06dcbda1e33b9dec70770b2906e8f1b1601f8e"} Jul 22 16:22:21.383892 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.380583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-722k6" event={"ID":"2e056b28-ef60-4e85-8c47-f17c5e01ed35","Type":"ContainerStarted","Data":"2c23397bf16b992563deae1886d287e90c9e15cb345e2e228a6b48f31080dd1c"} Jul 22 16:22:21.383892 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.382272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:21.392137 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.392099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v89rw" event={"ID":"bff7c473-0eea-4e23-85f8-0e7162fe924c","Type":"ContainerStarted","Data":"03b6f8563e3f3dfc24c4d54c188698bb6209d5606a275d686b9f80e47f82665c"} Jul 22 16:22:21.393855 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.393679 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-722k6" Jul 22 16:22:21.397748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.397651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"6129ee0371820ec5ba9e96f9168a47d1ce8822050790ccb4960bac8ea77c3986"} Jul 22 16:22:21.401837 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.400109 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-722k6" podStartSLOduration=1.221867519 podStartE2EDuration="19.400091095s" totalImagesPullingTime="18.178223576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:21.399047933 +0000 UTC m=+98.979141834" watchObservedRunningTime="2026-07-22 16:22:21.400091095 +0000 UTC m=+98.980184993" Jul 22 16:22:21.405039 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.404967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"2e5dd521bb49185ae5cc20c6dccdb73e40731a8472fdeebc35270a6a6e177a7e"} Jul 22 16:22:21.536237 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.536168 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcdfb8bb6-gpvgq"] Jul 22 16:22:21.544593 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:21.543902 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151e17e2_8a8d_49c3_80aa_44d395d7c47c.slice/crio-666f1f423b98ba88a8138ce6a8004ddbbe8fe78a50df1171f7eb1105f4cd32bb WatchSource:0}: Error finding container 666f1f423b98ba88a8138ce6a8004ddbbe8fe78a50df1171f7eb1105f4cd32bb: Status 404 returned error can't find the container with id 666f1f423b98ba88a8138ce6a8004ddbbe8fe78a50df1171f7eb1105f4cd32bb Jul 22 16:22:21.570229 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.570197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14523a9d-65dc-4eb8-ab71-be5833fa978d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:21.573952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.573922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14523a9d-65dc-4eb8-ab71-be5833fa978d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f49vh\" (UID: \"14523a9d-65dc-4eb8-ab71-be5833fa978d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:21.851787 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:21.851745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:22.079147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:22.079074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh"] Jul 22 16:22:22.240253 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:22.240215 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:22:22.416503 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:22.416424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcdfb8bb6-gpvgq" event={"ID":"151e17e2-8a8d-49c3-80aa-44d395d7c47c","Type":"ContainerStarted","Data":"666f1f423b98ba88a8138ce6a8004ddbbe8fe78a50df1171f7eb1105f4cd32bb"} Jul 22 16:22:23.841196 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:23.841153 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14523a9d_65dc_4eb8_ab71_be5833fa978d.slice/crio-a915677950a587decf14357a378dc192952376701fd77f0b319feb310fa45102 WatchSource:0}: Error finding container a915677950a587decf14357a378dc192952376701fd77f0b319feb310fa45102: Status 404 returned error can't find the container with id a915677950a587decf14357a378dc192952376701fd77f0b319feb310fa45102 Jul 22 16:22:24.432227 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:24.432184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" event={"ID":"14523a9d-65dc-4eb8-ab71-be5833fa978d","Type":"ContainerStarted","Data":"a915677950a587decf14357a378dc192952376701fd77f0b319feb310fa45102"} Jul 22 16:22:25.252251 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:25.252219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z6s8g" Jul 22 16:22:25.309696 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:25.309651 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c54c58fd-zhd9r" Jul 22 16:22:27.253792 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:27.253578 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" podUID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" containerName="registry" containerID="cri-o://423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727" gracePeriod=30 Jul 22 16:22:30.391425 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.391379 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:22:30.458653 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.458624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b"} Jul 22 16:22:30.460020 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.459998 2580 generic.go:356] "Generic (PLEG): container finished" podID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" containerID="423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727" exitCode=0 Jul 22 16:22:30.460112 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.460037 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" Jul 22 16:22:30.460112 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.460043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" event={"ID":"0c3656eb-3a29-4ed5-9748-676c2922e7d8","Type":"ContainerDied","Data":"423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727"} Jul 22 16:22:30.460112 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.460078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6495bccd87-68tbm" event={"ID":"0c3656eb-3a29-4ed5-9748-676c2922e7d8","Type":"ContainerDied","Data":"425dd676b3cf7c9f18ba1abd4ac9b6bbbd94c2d97c0bb8a61c39d7197faa8aa6"} Jul 22 16:22:30.460112 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.460107 2580 scope.go:117] "RemoveContainer" containerID="423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727" Jul 22 16:22:30.467704 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.467686 2580 scope.go:117] "RemoveContainer" containerID="423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727" Jul 22 16:22:30.468194 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:22:30.468171 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727\": container with ID starting with 423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727 not found: ID does not exist" containerID="423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727" Jul 22 16:22:30.468277 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.468199 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727"} err="failed to get container status \"423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727\": rpc error: code = NotFound desc = could not find container \"423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727\": container with ID starting with 423bbc2e970398cdeb1fbc9f41ae4cedfe6082cfc161af69e8af3a7c6e01a727 not found: ID does not exist" Jul 22 16:22:30.473664 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473648 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-trusted-ca\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473745 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-installation-pull-secrets\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473745 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473734 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-certificates\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473776 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r255j\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-kube-api-access-r255j\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473792 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-bound-sa-token\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473831 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473860 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c3656eb-3a29-4ed5-9748-676c2922e7d8-ca-trust-extracted\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.473989 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.473890 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-image-registry-private-configuration\") pod \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\" (UID: \"0c3656eb-3a29-4ed5-9748-676c2922e7d8\") " Jul 22 16:22:30.474056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.474028 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:22:30.474318 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.474281 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:22:30.474434 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.474355 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-trusted-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.476530 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.476454 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:22:30.476615 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.476570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:22:30.476678 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.476622 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:22:30.476725 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.476673 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:22:30.476902 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.476883 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-kube-api-access-r255j" (OuterVolumeSpecName: "kube-api-access-r255j") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "kube-api-access-r255j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:22:30.486140 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.486090 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3656eb-3a29-4ed5-9748-676c2922e7d8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0c3656eb-3a29-4ed5-9748-676c2922e7d8" (UID: "0c3656eb-3a29-4ed5-9748-676c2922e7d8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:22:30.575862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575836 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-installation-pull-secrets\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.575862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575864 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-certificates\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.576032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575873 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r255j\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-kube-api-access-r255j\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.576032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575887 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-bound-sa-token\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.576032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575897 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c3656eb-3a29-4ed5-9748-676c2922e7d8-registry-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.576032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575905 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c3656eb-3a29-4ed5-9748-676c2922e7d8-ca-trust-extracted\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.576032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.575914 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c3656eb-3a29-4ed5-9748-676c2922e7d8-image-registry-private-configuration\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:22:30.782184 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.782158 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6495bccd87-68tbm"] Jul 22 16:22:30.786875 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.786857 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6495bccd87-68tbm"] Jul 22 16:22:30.978950 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:30.978918 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" path="/var/lib/kubelet/pods/0c3656eb-3a29-4ed5-9748-676c2922e7d8/volumes" Jul 22 16:22:31.470903 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.470870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"5d85ef09d161469dc946201371cb744d83f3e5332b5d58e3de61905883e16bbc"} Jul 22 16:22:31.471345 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.470972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"b979a5d2acd9b80a7a9df6c07c6fd818e33810b831c92ae25c33654d72b258d6"} Jul 22 16:22:31.471345 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.471011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"e80328f3f55922ccd645efabe9f197983d7442acfb9ba5395c7b4b6a3b2c0169"} Jul 22 16:22:31.472311 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.472286 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b" exitCode=0 Jul 22 16:22:31.472431 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.472370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b"} Jul 22 16:22:31.473774 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.473752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcdfb8bb6-gpvgq" event={"ID":"151e17e2-8a8d-49c3-80aa-44d395d7c47c","Type":"ContainerStarted","Data":"efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db"} Jul 22 16:22:31.476505 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.476485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v89rw" event={"ID":"bff7c473-0eea-4e23-85f8-0e7162fe924c","Type":"ContainerStarted","Data":"60dedb3747eb4f40624a70bc9aacd5c2a0ddc4383305f23f8ea47fb806e7d0c5"} Jul 22 16:22:31.477960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.477938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" event={"ID":"14523a9d-65dc-4eb8-ab71-be5833fa978d","Type":"ContainerStarted","Data":"203aba03d7cf8bdbee9d2e4587bb757f3112bd6c276cdedc37271502ea83c804"} Jul 22 16:22:31.478145 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.478124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:31.479473 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.479448 2580 generic.go:356] "Generic (PLEG): container finished" podID="c6162922-fec2-43bf-a829-85fdf6b54207" containerID="a6d3d176992095c4c07a2d7a9b58184154e9bea86e2597c1bf9b0e9de6d1a561" exitCode=0 Jul 22 16:22:31.479566 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.479527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8x8ks" event={"ID":"c6162922-fec2-43bf-a829-85fdf6b54207","Type":"ContainerDied","Data":"a6d3d176992095c4c07a2d7a9b58184154e9bea86e2597c1bf9b0e9de6d1a561"} Jul 22 16:22:31.481743 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.481721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" event={"ID":"48b032e6-40fc-418e-9041-1e0928a33eec","Type":"ContainerStarted","Data":"5c5e317e66c0bdc0a59286f26e5c84b8782243defe9f864878191c4ee7f4676c"} Jul 22 16:22:31.483236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.483215 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" Jul 22 16:22:31.518563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.518510 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v89rw" podStartSLOduration=33.88268942 podStartE2EDuration="1m0.518490724s" totalImagesPullingTime="26.635801304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:21:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:31.516632701 +0000 UTC m=+109.096726626" watchObservedRunningTime="2026-07-22 16:22:31.518490724 +0000 UTC m=+109.098584624" Jul 22 16:22:31.553716 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.553591 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bcdfb8bb6-gpvgq" podStartSLOduration=2.744367825 podStartE2EDuration="11.553570684s" totalImagesPullingTime="8.809202859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:31.55147547 +0000 UTC m=+109.131569400" watchObservedRunningTime="2026-07-22 16:22:31.553570684 +0000 UTC m=+109.133664580" Jul 22 16:22:31.569628 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.569584 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f49vh" podStartSLOduration=4.666656103 podStartE2EDuration="11.569570944s" totalImagesPullingTime="6.902914841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:31.567830575 +0000 UTC m=+109.147924477" watchObservedRunningTime="2026-07-22 16:22:31.569570944 +0000 UTC m=+109.149664840" Jul 22 16:22:31.588378 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:31.588328 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5t46n" podStartSLOduration=6.468255517 podStartE2EDuration="15.588306858s" totalImagesPullingTime="9.120051341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:31.587090456 +0000 UTC m=+109.167184355" watchObservedRunningTime="2026-07-22 16:22:31.588306858 +0000 UTC m=+109.168400756" Jul 22 16:22:32.489291 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:32.488743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8x8ks" event={"ID":"c6162922-fec2-43bf-a829-85fdf6b54207","Type":"ContainerStarted","Data":"c1089a70bf7a9a8b30797df3a1dc9100a4a1e294d8ca60119cf595c6591a00c5"} Jul 22 16:22:32.489291 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:32.488813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8x8ks" event={"ID":"c6162922-fec2-43bf-a829-85fdf6b54207","Type":"ContainerStarted","Data":"ecd9415fd48fd5ef8bd8860b0f68965a37273bdeb6708dbc92dedd5f78ec30c9"} Jul 22 16:22:32.492193 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:32.492166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"ae2ea39924943a99423186dce298e41179f9e8543a9bc63f26b96deab1a0646f"} Jul 22 16:22:32.492285 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:32.492242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"ff83b01fa08dd78ab163196235743d6924262c6a4d1ae93cff8b9c6d30f6e0ad"} Jul 22 16:22:32.508988 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:32.508926 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8x8ks" podStartSLOduration=6.9890318 podStartE2EDuration="16.508908507s" totalImagesPullingTime="9.430975926s" totalInitContainerRuntime="88.900781ms" isStatefulPod=true podCreationTimestamp="2026-07-22 16:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:32.507853805 +0000 UTC m=+110.087947702" watchObservedRunningTime="2026-07-22 16:22:32.508908507 +0000 UTC m=+110.089002404" Jul 22 16:22:33.499456 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:33.499410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" event={"ID":"d0def3e0-1f9b-4983-82aa-2cb760d79c32","Type":"ContainerStarted","Data":"622ceaade49a1d03d02eb3b9ff4016c469d6062b79591ce1c2d1c589203510c1"} Jul 22 16:22:33.499952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:33.499916 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:33.531434 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:33.531364 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" podStartSLOduration=4.972015587 podStartE2EDuration="15.531345381s" totalImagesPullingTime="10.559329794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:33.529117065 +0000 UTC m=+111.109210989" watchObservedRunningTime="2026-07-22 16:22:33.531345381 +0000 UTC m=+111.111439313" Jul 22 16:22:34.506413 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.506360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd"} Jul 22 16:22:34.506791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.506449 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546"} Jul 22 16:22:34.506791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.506484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0"} Jul 22 16:22:34.506791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.506514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58"} Jul 22 16:22:34.506791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.506543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3"} Jul 22 16:22:34.506791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.506574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerStarted","Data":"d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34"} Jul 22 16:22:34.533885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:34.533801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.412803679 podStartE2EDuration="17.533784231s" totalImagesPullingTime="11.107942927s" totalInitContainerRuntime="13.037625ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:34.531702137 +0000 UTC m=+112.111796071" watchObservedRunningTime="2026-07-22 16:22:34.533784231 +0000 UTC m=+112.113878127" Jul 22 16:22:39.512011 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:39.511981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-ffbd75db4-lnp9q" Jul 22 16:22:41.350699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:41.350668 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:41.351064 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:41.350722 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:41.355369 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:41.355346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:41.540622 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:41.540596 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:22:51.869748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:51.869706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:22:51.871976 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:51.871953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79635de-e4ce-4ebc-9921-b7e89ccd30bb-metrics-certs\") pod \"network-metrics-daemon-kfn8q\" (UID: \"d79635de-e4ce-4ebc-9921-b7e89ccd30bb\") " pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:22:51.892474 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:51.892445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9mc4s\"" Jul 22 16:22:51.900521 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:51.900503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfn8q" Jul 22 16:22:52.072408 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:52.072372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfn8q"] Jul 22 16:22:52.076237 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:22:52.076199 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79635de_e4ce_4ebc_9921_b7e89ccd30bb.slice/crio-a499c56143a5f7b893ec11bc195b1a9cdb07025997ec8bcd7d09e0ec3b392f91 WatchSource:0}: Error finding container a499c56143a5f7b893ec11bc195b1a9cdb07025997ec8bcd7d09e0ec3b392f91: Status 404 returned error can't find the container with id a499c56143a5f7b893ec11bc195b1a9cdb07025997ec8bcd7d09e0ec3b392f91 Jul 22 16:22:52.582034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:52.581992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfn8q" event={"ID":"d79635de-e4ce-4ebc-9921-b7e89ccd30bb","Type":"ContainerStarted","Data":"a499c56143a5f7b893ec11bc195b1a9cdb07025997ec8bcd7d09e0ec3b392f91"} Jul 22 16:22:53.587874 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:53.587841 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c3cd233-f901-441f-ba35-df1a4a9e9c09" containerID="47cb7a69d63ffd23156453d22be67bb9b5df85ca19577c215bab54207dec0da3" exitCode=0 Jul 22 16:22:53.588350 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:53.587897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" event={"ID":"4c3cd233-f901-441f-ba35-df1a4a9e9c09","Type":"ContainerDied","Data":"47cb7a69d63ffd23156453d22be67bb9b5df85ca19577c215bab54207dec0da3"} Jul 22 16:22:53.588350 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:53.588286 2580 scope.go:117] "RemoveContainer" containerID="47cb7a69d63ffd23156453d22be67bb9b5df85ca19577c215bab54207dec0da3" Jul 22 16:22:54.593600 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:54.593561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfn8q" event={"ID":"d79635de-e4ce-4ebc-9921-b7e89ccd30bb","Type":"ContainerStarted","Data":"483671a8ced44bec5c6253cef7d0b71c1395a79c3f91c5e82890821012265130"} Jul 22 16:22:54.594042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:54.593648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfn8q" event={"ID":"d79635de-e4ce-4ebc-9921-b7e89ccd30bb","Type":"ContainerStarted","Data":"9a9c7ae52634be9d6c49e48bee5110cecff5f88e8da8e2eb864ebcc7c4d5e2f7"} Jul 22 16:22:54.595327 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:54.595306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-szkq2" event={"ID":"4c3cd233-f901-441f-ba35-df1a4a9e9c09","Type":"ContainerStarted","Data":"46c2c1479fef2b56dab5a86b803026eb507604c7f00599b2ab09327ec0e23b13"} Jul 22 16:22:54.611441 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:54.611381 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kfn8q" podStartSLOduration=130.079195422 podStartE2EDuration="2m11.611368386s" totalImagesPullingTime="1.532172964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:22:54.610309333 +0000 UTC m=+132.190403231" watchObservedRunningTime="2026-07-22 16:22:54.611368386 +0000 UTC m=+132.191462351" Jul 22 16:22:56.604381 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:56.604348 2580 generic.go:356] "Generic (PLEG): container finished" podID="e78842c1-409e-4a48-a689-901e7f172152" containerID="e72a26245d15d2e329e14df7d0d61bbbf0ad67854ebfb728c13c237615f23e96" exitCode=0 Jul 22 16:22:56.604782 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:56.604420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" event={"ID":"e78842c1-409e-4a48-a689-901e7f172152","Type":"ContainerDied","Data":"e72a26245d15d2e329e14df7d0d61bbbf0ad67854ebfb728c13c237615f23e96"} Jul 22 16:22:56.604782 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:56.604771 2580 scope.go:117] "RemoveContainer" containerID="e72a26245d15d2e329e14df7d0d61bbbf0ad67854ebfb728c13c237615f23e96" Jul 22 16:22:57.609588 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:22:57.609555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-p9kbk" event={"ID":"e78842c1-409e-4a48-a689-901e7f172152","Type":"ContainerStarted","Data":"dce876a4b8b37f6b9bcb935600b560910b8514d88f4f37c69164632c6c805eb7"} Jul 22 16:23:36.412010 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.411927 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:23:36.412534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.412208 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="alertmanager" containerID="cri-o://d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34" gracePeriod=120 Jul 22 16:23:36.412534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.412288 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-web" containerID="cri-o://36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58" gracePeriod=120 Jul 22 16:23:36.412534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.412285 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-metric" containerID="cri-o://a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546" gracePeriod=120 Jul 22 16:23:36.412534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.412324 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy" containerID="cri-o://5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0" gracePeriod=120 Jul 22 16:23:36.412534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.412370 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="config-reloader" containerID="cri-o://611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3" gracePeriod=120 Jul 22 16:23:36.412534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.412385 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="prom-label-proxy" containerID="cri-o://460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd" gracePeriod=120 Jul 22 16:23:36.780599 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780568 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd" exitCode=0 Jul 22 16:23:36.780599 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780593 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546" exitCode=0 Jul 22 16:23:36.780599 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780600 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0" exitCode=0 Jul 22 16:23:36.780599 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780606 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3" exitCode=0 Jul 22 16:23:36.780828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780612 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34" exitCode=0 Jul 22 16:23:36.780828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd"} Jul 22 16:23:36.780828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546"} Jul 22 16:23:36.780828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780726 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0"} Jul 22 16:23:36.780828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3"} Jul 22 16:23:36.780828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:36.780775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34"} Jul 22 16:23:37.658760 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.658738 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.746458 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746359 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-web\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746458 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746420 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-web-config\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746458 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746447 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-trusted-ca-bundle\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746470 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-tls-assets\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746495 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-main-db\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746511 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-cluster-tls-config\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-out\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746574 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx2gz\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-kube-api-access-zx2gz\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746598 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-volume\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746672 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.746723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746713 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-metrics-client-ca\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.747156 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746748 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-main-tls\") pod \"6a11e974-b17c-43b7-83fb-c0c5b0027710\" (UID: \"6a11e974-b17c-43b7-83fb-c0c5b0027710\") " Jul 22 16:23:37.747156 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746838 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:23:37.747156 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.746892 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:23:37.747156 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.747142 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.747365 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.747166 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-alertmanager-main-db\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.747365 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.747233 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:23:37.749318 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.749288 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:23:37.749449 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.749378 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.749977 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.749940 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-volume" (OuterVolumeSpecName: "config-volume") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.750203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.750178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-out" (OuterVolumeSpecName: "config-out") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:23:37.750279 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.750226 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-kube-api-access-zx2gz" (OuterVolumeSpecName: "kube-api-access-zx2gz") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "kube-api-access-zx2gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:23:37.750443 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.750424 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.751275 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.751248 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.751357 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.751312 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.753840 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.753819 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.760279 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.760254 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-web-config" (OuterVolumeSpecName: "web-config") pod "6a11e974-b17c-43b7-83fb-c0c5b0027710" (UID: "6a11e974-b17c-43b7-83fb-c0c5b0027710"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:23:37.788154 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.788132 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerID="36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58" exitCode=0 Jul 22 16:23:37.788230 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.788185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58"} Jul 22 16:23:37.788230 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.788205 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.788303 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.788242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a11e974-b17c-43b7-83fb-c0c5b0027710","Type":"ContainerDied","Data":"2e5dd521bb49185ae5cc20c6dccdb73e40731a8472fdeebc35270a6a6e177a7e"} Jul 22 16:23:37.788303 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.788276 2580 scope.go:117] "RemoveContainer" containerID="460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd" Jul 22 16:23:37.795544 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.795525 2580 scope.go:117] "RemoveContainer" containerID="a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546" Jul 22 16:23:37.802470 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.802452 2580 scope.go:117] "RemoveContainer" containerID="5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0" Jul 22 16:23:37.808752 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.808732 2580 scope.go:117] "RemoveContainer" containerID="36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58" Jul 22 16:23:37.817026 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.817006 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:23:37.819067 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.819035 2580 scope.go:117] "RemoveContainer" containerID="611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3" Jul 22 16:23:37.820283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.820265 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:23:37.825931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.825897 2580 scope.go:117] "RemoveContainer" containerID="d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34" Jul 22 16:23:37.832895 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.832879 2580 scope.go:117] "RemoveContainer" containerID="c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b" Jul 22 16:23:37.839255 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.839237 2580 scope.go:117] "RemoveContainer" containerID="460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd" Jul 22 16:23:37.839502 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.839484 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd\": container with ID starting with 460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd not found: ID does not exist" containerID="460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd" Jul 22 16:23:37.839554 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.839508 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd"} err="failed to get container status \"460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd\": rpc error: code = NotFound desc = could not find container \"460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd\": container with ID starting with 460e0aca237267ae9f8ea3162f928daa32c9e495f0f499fe0f0276c2cd1550dd not found: ID does not exist" Jul 22 16:23:37.839554 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.839527 2580 scope.go:117] "RemoveContainer" containerID="a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546" Jul 22 16:23:37.839747 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.839728 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546\": container with ID starting with a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546 not found: ID does not exist" containerID="a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546" Jul 22 16:23:37.839803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.839756 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546"} err="failed to get container status \"a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546\": rpc error: code = NotFound desc = could not find container \"a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546\": container with ID starting with a67efa386047432ce18dfd4ab53acfe4cc768266ead3a15c62ad08b0a5b2f546 not found: ID does not exist" Jul 22 16:23:37.839803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.839780 2580 scope.go:117] "RemoveContainer" containerID="5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0" Jul 22 16:23:37.839988 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.839972 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0\": container with ID starting with 5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0 not found: ID does not exist" containerID="5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0" Jul 22 16:23:37.840054 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.839994 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0"} err="failed to get container status \"5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0\": rpc error: code = NotFound desc = could not find container \"5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0\": container with ID starting with 5aa6f26d47a995ee0884d58b786cb09dff554b7a1e26f202e5c3bc77307d9dc0 not found: ID does not exist" Jul 22 16:23:37.840054 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840012 2580 scope.go:117] "RemoveContainer" containerID="36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58" Jul 22 16:23:37.840216 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.840198 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58\": container with ID starting with 36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58 not found: ID does not exist" containerID="36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58" Jul 22 16:23:37.840259 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840220 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58"} err="failed to get container status \"36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58\": rpc error: code = NotFound desc = could not find container \"36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58\": container with ID starting with 36c5889d10d8d1924cb626a0d2edfb321801ec2a55d578d9db52bd314143dd58 not found: ID does not exist" Jul 22 16:23:37.840259 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840236 2580 scope.go:117] "RemoveContainer" containerID="611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3" Jul 22 16:23:37.840476 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.840458 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3\": container with ID starting with 611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3 not found: ID does not exist" containerID="611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3" Jul 22 16:23:37.840540 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840482 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3"} err="failed to get container status \"611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3\": rpc error: code = NotFound desc = could not find container \"611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3\": container with ID starting with 611b8e083fb00ce64d1f7b619d91d9b25449f45e7625074028176b0bc51b8be3 not found: ID does not exist" Jul 22 16:23:37.840540 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840499 2580 scope.go:117] "RemoveContainer" containerID="d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34" Jul 22 16:23:37.840699 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.840684 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34\": container with ID starting with d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34 not found: ID does not exist" containerID="d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34" Jul 22 16:23:37.840737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840703 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34"} err="failed to get container status \"d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34\": rpc error: code = NotFound desc = could not find container \"d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34\": container with ID starting with d27a939e2a5b278f253aac6c1730ea36358b4a01931f762444f4cd20d5eced34 not found: ID does not exist" Jul 22 16:23:37.840737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840716 2580 scope.go:117] "RemoveContainer" containerID="c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b" Jul 22 16:23:37.840936 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:23:37.840915 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b\": container with ID starting with c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b not found: ID does not exist" containerID="c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b" Jul 22 16:23:37.841011 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.840935 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b"} err="failed to get container status \"c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b\": rpc error: code = NotFound desc = could not find container \"c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b\": container with ID starting with c58cf3ac39901036b08da624ded1b04ed4449d34cec5ba391753e3b50920379b not found: ID does not exist" Jul 22 16:23:37.848302 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848280 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-main-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848412 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848307 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848412 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848346 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-web-config\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848412 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848361 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-tls-assets\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848412 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848374 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-cluster-tls-config\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848412 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848386 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-out\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848413 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2gz\" (UniqueName: \"kubernetes.io/projected/6a11e974-b17c-43b7-83fb-c0c5b0027710-kube-api-access-zx2gz\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848427 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848441 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-config-volume\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848455 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a11e974-b17c-43b7-83fb-c0c5b0027710-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.848597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.848469 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a11e974-b17c-43b7-83fb-c0c5b0027710-metrics-client-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:23:37.851635 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851616 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:23:37.851944 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851931 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="alertmanager" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851946 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="alertmanager" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851957 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851962 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851971 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="config-reloader" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851977 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="config-reloader" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851984 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="prom-label-proxy" Jul 22 16:23:37.851993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.851990 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="prom-label-proxy" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852001 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-metric" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852007 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-metric" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852017 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-web" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852023 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-web" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852030 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" containerName="registry" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852035 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" containerName="registry" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852041 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="init-config-reloader" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852046 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="init-config-reloader" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852097 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="prom-label-proxy" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852106 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="config-reloader" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852113 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852119 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-metric" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852124 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3656eb-3a29-4ed5-9748-676c2922e7d8" containerName="registry" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852130 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="alertmanager" Jul 22 16:23:37.852203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.852135 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" containerName="kube-rbac-proxy-web" Jul 22 16:23:37.857052 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.857035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.859209 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859190 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 16:23:37.859536 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 16:23:37.859536 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 16:23:37.859536 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 16:23:37.859733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859559 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dwzqb\"" Jul 22 16:23:37.859733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859511 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 16:23:37.859733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 16:23:37.859879 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859770 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 16:23:37.859945 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.859926 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 16:23:37.863932 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.863911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 16:23:37.869642 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.869622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:23:37.949083 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f716828-080d-4e83-b272-17d664e7a8eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949245 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f716828-080d-4e83-b272-17d664e7a8eb-config-out\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949245 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949245 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f716828-080d-4e83-b272-17d664e7a8eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949245 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f716828-080d-4e83-b272-17d664e7a8eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949245 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8v46\" (UniqueName: \"kubernetes.io/projected/7f716828-080d-4e83-b272-17d664e7a8eb-kube-api-access-d8v46\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f716828-080d-4e83-b272-17d664e7a8eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-web-config\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949557 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949557 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:37.949557 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:37.949489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050584 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050584 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050584 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050584 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050876 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f716828-080d-4e83-b272-17d664e7a8eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050876 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f716828-080d-4e83-b272-17d664e7a8eb-config-out\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050876 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.050876 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f716828-080d-4e83-b272-17d664e7a8eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.051069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f716828-080d-4e83-b272-17d664e7a8eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.051069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8v46\" (UniqueName: \"kubernetes.io/projected/7f716828-080d-4e83-b272-17d664e7a8eb-kube-api-access-d8v46\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.051069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.051069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.050972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f716828-080d-4e83-b272-17d664e7a8eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.051069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.051007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-web-config\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.051318 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.051126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f716828-080d-4e83-b272-17d664e7a8eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.052198 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.052132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f716828-080d-4e83-b272-17d664e7a8eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.052686 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.052637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f716828-080d-4e83-b272-17d664e7a8eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.053531 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.053507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f716828-080d-4e83-b272-17d664e7a8eb-config-out\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.053779 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.053752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.053883 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.053808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.053953 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.053880 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-web-config\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.054060 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.054042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.054144 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.054124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.054865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.054849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.055191 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.055174 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f716828-080d-4e83-b272-17d664e7a8eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.055525 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.055508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f716828-080d-4e83-b272-17d664e7a8eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.061540 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.061523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8v46\" (UniqueName: \"kubernetes.io/projected/7f716828-080d-4e83-b272-17d664e7a8eb-kube-api-access-d8v46\") pod \"alertmanager-main-0\" (UID: \"7f716828-080d-4e83-b272-17d664e7a8eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.166925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.166892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:23:38.304494 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.304470 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:23:38.306309 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:23:38.306285 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f716828_080d_4e83_b272_17d664e7a8eb.slice/crio-6d78911e5ec3ba38ccd755678850e92f0a88908b028429fa10aca88aa85f1e46 WatchSource:0}: Error finding container 6d78911e5ec3ba38ccd755678850e92f0a88908b028429fa10aca88aa85f1e46: Status 404 returned error can't find the container with id 6d78911e5ec3ba38ccd755678850e92f0a88908b028429fa10aca88aa85f1e46 Jul 22 16:23:38.794796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.794767 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f716828-080d-4e83-b272-17d664e7a8eb" containerID="1179700664b802f4860238809e934a89157fb1d25e6b123ad8c942a0f3068650" exitCode=0 Jul 22 16:23:38.795180 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.794840 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerDied","Data":"1179700664b802f4860238809e934a89157fb1d25e6b123ad8c942a0f3068650"} Jul 22 16:23:38.795180 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.794888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"6d78911e5ec3ba38ccd755678850e92f0a88908b028429fa10aca88aa85f1e46"} Jul 22 16:23:38.978645 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:38.978614 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a11e974-b17c-43b7-83fb-c0c5b0027710" path="/var/lib/kubelet/pods/6a11e974-b17c-43b7-83fb-c0c5b0027710/volumes" Jul 22 16:23:39.801240 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.801199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"ccbc727a06aade0be44060be3e1603bcc4fab10a682fd71eef2b2d79cf05ebf7"} Jul 22 16:23:39.801620 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.801317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"ab9cee8bad7030081877e0465222812e6219db362dfbe35e0536d7ba1d3944e2"} Jul 22 16:23:39.801620 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.801358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"1c2f1da5bbe5de3029d801c2105e2eb0468276b3b68911de688775f8a52720b3"} Jul 22 16:23:39.801620 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.801410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"8953dfeae1056dbb0da802d497ac832af995b0373370071a44e4b6b23b0f9027"} Jul 22 16:23:39.801620 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.801447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"0e19d4ed96170e2acc4ee0d5307c01f32d0720a692318f3b774b635b65f7a076"} Jul 22 16:23:39.801620 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.801483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f716828-080d-4e83-b272-17d664e7a8eb","Type":"ContainerStarted","Data":"c16efea9dc5fcd7de47ba926933d69b2740990ceac3f01362dc2505463e9a747"} Jul 22 16:23:39.833545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:39.833474 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.819489249 podStartE2EDuration="2.833454738s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.965489ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:23:39.832459004 +0000 UTC m=+177.412552912" watchObservedRunningTime="2026-07-22 16:23:39.833454738 +0000 UTC m=+177.413548636" Jul 22 16:23:40.507513 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.507482 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz"] Jul 22 16:23:40.510963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.510946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.525229 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.525211 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 16:23:40.526277 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.526260 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 16:23:40.526376 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.526293 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 16:23:40.526376 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.526372 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 16:23:40.526472 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.526441 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 16:23:40.527507 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.527489 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7wc9p\"" Jul 22 16:23:40.530284 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.530267 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 16:23:40.545763 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.545736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz"] Jul 22 16:23:40.674545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-serving-certs-ca-bundle\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-federate-client-tls\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-secret-telemeter-client\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxnc\" (UniqueName: \"kubernetes.io/projected/7b76e213-2523-49ea-8743-320cbde976a0-kube-api-access-xlxnc\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-telemeter-client-tls\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.674963 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.674889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-metrics-client-ca\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.775756 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-telemeter-client-tls\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.775756 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.775756 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-metrics-client-ca\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-serving-certs-ca-bundle\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-federate-client-tls\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-secret-telemeter-client\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.775929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxnc\" (UniqueName: \"kubernetes.io/projected/7b76e213-2523-49ea-8743-320cbde976a0-kube-api-access-xlxnc\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776760 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.776702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-metrics-client-ca\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.776917 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.776757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.777222 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.777194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b76e213-2523-49ea-8743-320cbde976a0-serving-certs-ca-bundle\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.778469 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.778439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.778573 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.778495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-telemeter-client-tls\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.778573 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.778566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-secret-telemeter-client\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.778692 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.778597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b76e213-2523-49ea-8743-320cbde976a0-federate-client-tls\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.787052 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.787029 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxnc\" (UniqueName: \"kubernetes.io/projected/7b76e213-2523-49ea-8743-320cbde976a0-kube-api-access-xlxnc\") pod \"telemeter-client-5b9cdf7b7c-jrtwz\" (UID: \"7b76e213-2523-49ea-8743-320cbde976a0\") " pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.820740 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.820711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" Jul 22 16:23:40.965698 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:40.965671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz"] Jul 22 16:23:40.967719 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:23:40.967679 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b76e213_2523_49ea_8743_320cbde976a0.slice/crio-3bf12f199d1104d8853c3c5ec8d7d6e709bb47c7ce8314845b42df94c2fafb6a WatchSource:0}: Error finding container 3bf12f199d1104d8853c3c5ec8d7d6e709bb47c7ce8314845b42df94c2fafb6a: Status 404 returned error can't find the container with id 3bf12f199d1104d8853c3c5ec8d7d6e709bb47c7ce8314845b42df94c2fafb6a Jul 22 16:23:41.811594 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:41.811550 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" event={"ID":"7b76e213-2523-49ea-8743-320cbde976a0","Type":"ContainerStarted","Data":"3bf12f199d1104d8853c3c5ec8d7d6e709bb47c7ce8314845b42df94c2fafb6a"} Jul 22 16:23:42.821912 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:42.821821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" event={"ID":"7b76e213-2523-49ea-8743-320cbde976a0","Type":"ContainerStarted","Data":"b6e5a2234aeb5fba26c785362650f8de1ad2997920614b8648de85a9894f322b"} Jul 22 16:23:42.821912 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:42.821907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" event={"ID":"7b76e213-2523-49ea-8743-320cbde976a0","Type":"ContainerStarted","Data":"23de062cb1d4ba2ccfcde2aaf444019176d32796d5bc925e9cc3c14d76d4818a"} Jul 22 16:23:42.822276 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:42.821927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" event={"ID":"7b76e213-2523-49ea-8743-320cbde976a0","Type":"ContainerStarted","Data":"f474606e7d86f6d8a25e47e08fa962d3ab7545aa5e1e02e616f6f4fc8062c03b"} Jul 22 16:23:42.857742 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:42.857692 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b9cdf7b7c-jrtwz" podStartSLOduration=1.253657852 podStartE2EDuration="2.857675628s" totalImagesPullingTime="1.604017776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:23:42.856236774 +0000 UTC m=+180.436330683" watchObservedRunningTime="2026-07-22 16:23:42.857675628 +0000 UTC m=+180.437769528" Jul 22 16:23:53.980234 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:23:53.980178 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcdfb8bb6-gpvgq"] Jul 22 16:24:19.001465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.001420 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bcdfb8bb6-gpvgq" podUID="151e17e2-8a8d-49c3-80aa-44d395d7c47c" containerName="console" containerID="cri-o://efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db" gracePeriod=15 Jul 22 16:24:19.248924 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.248896 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:24:19.271733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271665 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-oauth-serving-cert\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.271733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271702 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-trusted-ca-bundle\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.271906 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271755 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-service-ca\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.271906 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271791 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-oauth-config\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.271906 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271842 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-config\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.271906 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbx4c\" (UniqueName: \"kubernetes.io/projected/151e17e2-8a8d-49c3-80aa-44d395d7c47c-kube-api-access-sbx4c\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.272117 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.271923 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-serving-cert\") pod \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\" (UID: \"151e17e2-8a8d-49c3-80aa-44d395d7c47c\") " Jul 22 16:24:19.272117 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.272046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:24:19.272231 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.272132 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:24:19.272344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.272245 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-config" (OuterVolumeSpecName: "console-config") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:24:19.272344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.272302 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-oauth-serving-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.272344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.272324 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-trusted-ca-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.272648 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.272623 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-service-ca" (OuterVolumeSpecName: "service-ca") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:24:19.274170 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.274114 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151e17e2-8a8d-49c3-80aa-44d395d7c47c-kube-api-access-sbx4c" (OuterVolumeSpecName: "kube-api-access-sbx4c") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "kube-api-access-sbx4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:24:19.274283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.274219 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:19.274283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.274234 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "151e17e2-8a8d-49c3-80aa-44d395d7c47c" (UID: "151e17e2-8a8d-49c3-80aa-44d395d7c47c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:19.372754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.372714 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.372754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.372745 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-oauth-config\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.372754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.372756 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-config\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.372754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.372765 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbx4c\" (UniqueName: \"kubernetes.io/projected/151e17e2-8a8d-49c3-80aa-44d395d7c47c-kube-api-access-sbx4c\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.373017 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.372773 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/151e17e2-8a8d-49c3-80aa-44d395d7c47c-console-serving-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:24:19.974703 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.974668 2580 generic.go:356] "Generic (PLEG): container finished" podID="151e17e2-8a8d-49c3-80aa-44d395d7c47c" containerID="efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db" exitCode=0 Jul 22 16:24:19.974703 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.974701 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcdfb8bb6-gpvgq" Jul 22 16:24:19.974927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.974747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcdfb8bb6-gpvgq" event={"ID":"151e17e2-8a8d-49c3-80aa-44d395d7c47c","Type":"ContainerDied","Data":"efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db"} Jul 22 16:24:19.974927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.974786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcdfb8bb6-gpvgq" event={"ID":"151e17e2-8a8d-49c3-80aa-44d395d7c47c","Type":"ContainerDied","Data":"666f1f423b98ba88a8138ce6a8004ddbbe8fe78a50df1171f7eb1105f4cd32bb"} Jul 22 16:24:19.974927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.974806 2580 scope.go:117] "RemoveContainer" containerID="efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db" Jul 22 16:24:19.983591 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.983574 2580 scope.go:117] "RemoveContainer" containerID="efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db" Jul 22 16:24:19.983856 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:24:19.983836 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db\": container with ID starting with efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db not found: ID does not exist" containerID="efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db" Jul 22 16:24:19.983898 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.983864 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db"} err="failed to get container status \"efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db\": rpc error: code = NotFound desc = could not find container \"efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db\": container with ID starting with efc0938c3d78ab2cd04f743347f53b2be41f7ff4df51b50898bdda3a858611db not found: ID does not exist" Jul 22 16:24:19.995733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:19.995708 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcdfb8bb6-gpvgq"] Jul 22 16:24:20.001544 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:20.001525 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bcdfb8bb6-gpvgq"] Jul 22 16:24:20.978326 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:20.978286 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="151e17e2-8a8d-49c3-80aa-44d395d7c47c" path="/var/lib/kubelet/pods/151e17e2-8a8d-49c3-80aa-44d395d7c47c/volumes" Jul 22 16:24:54.688684 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.688649 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59685fb89b-vlj8s"] Jul 22 16:24:54.689124 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.689002 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151e17e2-8a8d-49c3-80aa-44d395d7c47c" containerName="console" Jul 22 16:24:54.689124 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.689015 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="151e17e2-8a8d-49c3-80aa-44d395d7c47c" containerName="console" Jul 22 16:24:54.689124 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.689088 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="151e17e2-8a8d-49c3-80aa-44d395d7c47c" containerName="console" Jul 22 16:24:54.692207 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.692191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.694650 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.694631 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bmnj5\"" Jul 22 16:24:54.695423 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.695374 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 16:24:54.695423 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.695411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 16:24:54.695680 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.695660 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 16:24:54.695825 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.695723 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 16:24:54.695900 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.695738 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 16:24:54.700256 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.700236 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 16:24:54.706329 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.706308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59685fb89b-vlj8s"] Jul 22 16:24:54.871659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288ll\" (UniqueName: \"kubernetes.io/projected/b7888501-0547-435e-9d8f-115c803e7aa1-kube-api-access-288ll\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.871659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-trusted-ca-bundle\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.871860 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-oauth-serving-cert\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.871860 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-console-config\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.871860 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-oauth-config\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.871860 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-service-ca\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.871985 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.871876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-serving-cert\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973043 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.972958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-oauth-serving-cert\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973043 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-console-config\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973043 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-oauth-config\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-service-ca\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-serving-cert\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288ll\" (UniqueName: \"kubernetes.io/projected/b7888501-0547-435e-9d8f-115c803e7aa1-kube-api-access-288ll\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-trusted-ca-bundle\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.973864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-oauth-serving-cert\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.974051 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-service-ca\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.974140 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.973855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-console-config\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.974140 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.974113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-trusted-ca-bundle\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.976139 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.976121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-serving-cert\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.976217 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.976161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-oauth-config\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:54.982718 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:54.982695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-288ll\" (UniqueName: \"kubernetes.io/projected/b7888501-0547-435e-9d8f-115c803e7aa1-kube-api-access-288ll\") pod \"console-59685fb89b-vlj8s\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:55.001329 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:55.001306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:24:55.131681 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:55.131650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59685fb89b-vlj8s"] Jul 22 16:24:55.135899 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:24:55.135873 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7888501_0547_435e_9d8f_115c803e7aa1.slice/crio-47c79d6fbb97a4b78a4792f5924c4262633856c160da319e5322e1612aa71ee8 WatchSource:0}: Error finding container 47c79d6fbb97a4b78a4792f5924c4262633856c160da319e5322e1612aa71ee8: Status 404 returned error can't find the container with id 47c79d6fbb97a4b78a4792f5924c4262633856c160da319e5322e1612aa71ee8 Jul 22 16:24:56.109865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:56.109827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59685fb89b-vlj8s" event={"ID":"b7888501-0547-435e-9d8f-115c803e7aa1","Type":"ContainerStarted","Data":"67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74"} Jul 22 16:24:56.110252 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:56.109881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59685fb89b-vlj8s" event={"ID":"b7888501-0547-435e-9d8f-115c803e7aa1","Type":"ContainerStarted","Data":"47c79d6fbb97a4b78a4792f5924c4262633856c160da319e5322e1612aa71ee8"} Jul 22 16:24:56.130452 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:24:56.130408 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59685fb89b-vlj8s" podStartSLOduration=2.1303795389999998 podStartE2EDuration="2.130379539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:24:56.128745713 +0000 UTC m=+253.708839623" watchObservedRunningTime="2026-07-22 16:24:56.130379539 +0000 UTC m=+253.710473435" Jul 22 16:25:05.002120 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:25:05.002088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:25:05.002529 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:25:05.002143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:25:05.006855 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:25:05.006831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:25:05.149287 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:25:05.149257 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:25:42.861723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:25:42.856322 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 16:26:42.461435 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.461383 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg"] Jul 22 16:26:42.464822 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.464806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.469700 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.469675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 16:26:42.469836 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.469677 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 16:26:42.470559 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.470542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gj5ff\"" Jul 22 16:26:42.476706 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.476682 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg"] Jul 22 16:26:42.549835 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.549798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.549835 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.549839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtf4\" (UniqueName: \"kubernetes.io/projected/f350e29b-b9a3-40aa-9460-39f6f7df32a7-kube-api-access-cgtf4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.550068 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.549892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.651368 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.651325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.651563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.651440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.651563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.651466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtf4\" (UniqueName: \"kubernetes.io/projected/f350e29b-b9a3-40aa-9460-39f6f7df32a7-kube-api-access-cgtf4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.651753 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.651730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.651823 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.651801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.663158 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.663133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtf4\" (UniqueName: \"kubernetes.io/projected/f350e29b-b9a3-40aa-9460-39f6f7df32a7-kube-api-access-cgtf4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.773500 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.773464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:26:42.913501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.913475 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg"] Jul 22 16:26:42.915961 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:26:42.915930 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf350e29b_b9a3_40aa_9460_39f6f7df32a7.slice/crio-3524f8fbc7435129e8e8106da8005a3ca22902467a0e2d15a691aee12b35c249 WatchSource:0}: Error finding container 3524f8fbc7435129e8e8106da8005a3ca22902467a0e2d15a691aee12b35c249: Status 404 returned error can't find the container with id 3524f8fbc7435129e8e8106da8005a3ca22902467a0e2d15a691aee12b35c249 Jul 22 16:26:42.917875 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:42.917856 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:26:43.532741 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:43.532701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" event={"ID":"f350e29b-b9a3-40aa-9460-39f6f7df32a7","Type":"ContainerStarted","Data":"3524f8fbc7435129e8e8106da8005a3ca22902467a0e2d15a691aee12b35c249"} Jul 22 16:26:50.563317 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:50.563280 2580 generic.go:356] "Generic (PLEG): container finished" podID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerID="1d35167c7012b899487fb799897f0db81656a00d768dfb10618a9f62db6c4bdf" exitCode=0 Jul 22 16:26:50.563810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:50.563335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" event={"ID":"f350e29b-b9a3-40aa-9460-39f6f7df32a7","Type":"ContainerDied","Data":"1d35167c7012b899487fb799897f0db81656a00d768dfb10618a9f62db6c4bdf"} Jul 22 16:26:53.578004 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:53.577970 2580 generic.go:356] "Generic (PLEG): container finished" podID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerID="59c8c2ff141b9a14c9b958fd7c5e78624ecd3c3b2b3a2c424e17acd934536b57" exitCode=0 Jul 22 16:26:53.578378 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:26:53.578054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" event={"ID":"f350e29b-b9a3-40aa-9460-39f6f7df32a7","Type":"ContainerDied","Data":"59c8c2ff141b9a14c9b958fd7c5e78624ecd3c3b2b3a2c424e17acd934536b57"} Jul 22 16:27:01.613903 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:01.613859 2580 generic.go:356] "Generic (PLEG): container finished" podID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerID="d1773c3684f31f4901434c2ad25bf951caafbccadcd91e26a38807b2ca1ef51a" exitCode=0 Jul 22 16:27:01.614258 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:01.613934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" event={"ID":"f350e29b-b9a3-40aa-9460-39f6f7df32a7","Type":"ContainerDied","Data":"d1773c3684f31f4901434c2ad25bf951caafbccadcd91e26a38807b2ca1ef51a"} Jul 22 16:27:02.740683 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.740659 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:27:02.833068 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.833028 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-util\") pod \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " Jul 22 16:27:02.833255 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.833086 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgtf4\" (UniqueName: \"kubernetes.io/projected/f350e29b-b9a3-40aa-9460-39f6f7df32a7-kube-api-access-cgtf4\") pod \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " Jul 22 16:27:02.833255 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.833145 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-bundle\") pod \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\" (UID: \"f350e29b-b9a3-40aa-9460-39f6f7df32a7\") " Jul 22 16:27:02.833891 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.833860 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-bundle" (OuterVolumeSpecName: "bundle") pod "f350e29b-b9a3-40aa-9460-39f6f7df32a7" (UID: "f350e29b-b9a3-40aa-9460-39f6f7df32a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:02.835336 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.835314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f350e29b-b9a3-40aa-9460-39f6f7df32a7-kube-api-access-cgtf4" (OuterVolumeSpecName: "kube-api-access-cgtf4") pod "f350e29b-b9a3-40aa-9460-39f6f7df32a7" (UID: "f350e29b-b9a3-40aa-9460-39f6f7df32a7"). InnerVolumeSpecName "kube-api-access-cgtf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:27:02.837819 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.837792 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-util" (OuterVolumeSpecName: "util") pod "f350e29b-b9a3-40aa-9460-39f6f7df32a7" (UID: "f350e29b-b9a3-40aa-9460-39f6f7df32a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:02.934338 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.934252 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:02.934338 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.934279 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtf4\" (UniqueName: \"kubernetes.io/projected/f350e29b-b9a3-40aa-9460-39f6f7df32a7-kube-api-access-cgtf4\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:02.934338 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:02.934294 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f350e29b-b9a3-40aa-9460-39f6f7df32a7-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:03.623925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:03.623891 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" event={"ID":"f350e29b-b9a3-40aa-9460-39f6f7df32a7","Type":"ContainerDied","Data":"3524f8fbc7435129e8e8106da8005a3ca22902467a0e2d15a691aee12b35c249"} Jul 22 16:27:03.623925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:03.623924 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404gpxg" Jul 22 16:27:03.624166 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:03.623939 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3524f8fbc7435129e8e8106da8005a3ca22902467a0e2d15a691aee12b35c249" Jul 22 16:27:10.489547 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489511 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn"] Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489840 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="extract" Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489853 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="extract" Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489867 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="util" Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489872 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="util" Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489882 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="pull" Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489901 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="pull" Jul 22 16:27:10.490016 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.489960 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f350e29b-b9a3-40aa-9460-39f6f7df32a7" containerName="extract" Jul 22 16:27:10.492487 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.492462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.496503 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.496469 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:27:10.496653 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.496631 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wxgvn\"" Jul 22 16:27:10.496714 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.496679 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 16:27:10.520708 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.520670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn"] Jul 22 16:27:10.599685 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.599656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qbn\" (UniqueName: \"kubernetes.io/projected/e4c36afd-b078-4106-851d-06776d9d5e63-kube-api-access-w7qbn\") pod \"cert-manager-operator-controller-manager-55c444bf86-5lpjn\" (UID: \"e4c36afd-b078-4106-851d-06776d9d5e63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.599820 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.599702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4c36afd-b078-4106-851d-06776d9d5e63-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5lpjn\" (UID: \"e4c36afd-b078-4106-851d-06776d9d5e63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.700585 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.700552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qbn\" (UniqueName: \"kubernetes.io/projected/e4c36afd-b078-4106-851d-06776d9d5e63-kube-api-access-w7qbn\") pod \"cert-manager-operator-controller-manager-55c444bf86-5lpjn\" (UID: \"e4c36afd-b078-4106-851d-06776d9d5e63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.700737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.700598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4c36afd-b078-4106-851d-06776d9d5e63-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5lpjn\" (UID: \"e4c36afd-b078-4106-851d-06776d9d5e63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.700935 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.700920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4c36afd-b078-4106-851d-06776d9d5e63-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5lpjn\" (UID: \"e4c36afd-b078-4106-851d-06776d9d5e63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.710368 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.710347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qbn\" (UniqueName: \"kubernetes.io/projected/e4c36afd-b078-4106-851d-06776d9d5e63-kube-api-access-w7qbn\") pod \"cert-manager-operator-controller-manager-55c444bf86-5lpjn\" (UID: \"e4c36afd-b078-4106-851d-06776d9d5e63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.801686 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.801602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" Jul 22 16:27:10.938044 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:10.938010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn"] Jul 22 16:27:10.942026 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:27:10.941997 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c36afd_b078_4106_851d_06776d9d5e63.slice/crio-ee58bed5ea7c383b21f0ccb422f4b676980e73628b776f94522d2c1129fca754 WatchSource:0}: Error finding container ee58bed5ea7c383b21f0ccb422f4b676980e73628b776f94522d2c1129fca754: Status 404 returned error can't find the container with id ee58bed5ea7c383b21f0ccb422f4b676980e73628b776f94522d2c1129fca754 Jul 22 16:27:11.658558 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:11.658526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" event={"ID":"e4c36afd-b078-4106-851d-06776d9d5e63","Type":"ContainerStarted","Data":"ee58bed5ea7c383b21f0ccb422f4b676980e73628b776f94522d2c1129fca754"} Jul 22 16:27:13.668642 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:13.668604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" event={"ID":"e4c36afd-b078-4106-851d-06776d9d5e63","Type":"ContainerStarted","Data":"04e779143084c3f1ace626994d853d205e14d77900861763bff9b41d27abe1e5"} Jul 22 16:27:13.693061 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:13.692819 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5lpjn" podStartSLOduration=1.757376201 podStartE2EDuration="3.69280427s" totalImagesPullingTime="1.935428069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:27:13.690563485 +0000 UTC m=+391.270657395" watchObservedRunningTime="2026-07-22 16:27:13.69280427 +0000 UTC m=+391.272898166" Jul 22 16:27:15.016022 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.015989 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh"] Jul 22 16:27:15.019710 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.019691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.023167 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.023145 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 16:27:15.023442 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.023421 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 16:27:15.024109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.024094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gj5ff\"" Jul 22 16:27:15.036414 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.036378 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh"] Jul 22 16:27:15.138783 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.138747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.138952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.138794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.138952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.138839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s6q\" (UniqueName: \"kubernetes.io/projected/01072ed1-780d-46ba-a1ae-e10c135c6551-kube-api-access-p9s6q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.239982 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.239954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s6q\" (UniqueName: \"kubernetes.io/projected/01072ed1-780d-46ba-a1ae-e10c135c6551-kube-api-access-p9s6q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.240168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.240003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.240168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.240036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.240413 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.240375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.240468 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.240434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.249849 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.249823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s6q\" (UniqueName: \"kubernetes.io/projected/01072ed1-780d-46ba-a1ae-e10c135c6551-kube-api-access-p9s6q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.329049 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.328973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:15.462998 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.462975 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh"] Jul 22 16:27:15.464816 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:27:15.464786 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01072ed1_780d_46ba_a1ae_e10c135c6551.slice/crio-43d7749f793f524799140c578e44223c0284cc4bf15029f025ef3d71251308b7 WatchSource:0}: Error finding container 43d7749f793f524799140c578e44223c0284cc4bf15029f025ef3d71251308b7: Status 404 returned error can't find the container with id 43d7749f793f524799140c578e44223c0284cc4bf15029f025ef3d71251308b7 Jul 22 16:27:15.678003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.677925 2580 generic.go:356] "Generic (PLEG): container finished" podID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerID="bf0dbe27368fc36ee4c31993450b02de420b2a065af6f1ef4c174f3bba5d93a8" exitCode=0 Jul 22 16:27:15.678147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.678016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" event={"ID":"01072ed1-780d-46ba-a1ae-e10c135c6551","Type":"ContainerDied","Data":"bf0dbe27368fc36ee4c31993450b02de420b2a065af6f1ef4c174f3bba5d93a8"} Jul 22 16:27:15.678147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:15.678074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" event={"ID":"01072ed1-780d-46ba-a1ae-e10c135c6551","Type":"ContainerStarted","Data":"43d7749f793f524799140c578e44223c0284cc4bf15029f025ef3d71251308b7"} Jul 22 16:27:16.769532 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.769499 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4"] Jul 22 16:27:16.773034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.773011 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:16.775809 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.775783 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 16:27:16.775931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.775790 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 16:27:16.775931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.775848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wphhn\"" Jul 22 16:27:16.783927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.783899 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4"] Jul 22 16:27:16.855545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.855511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8837b017-c001-4a7c-ae36-e582dd953e3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mcjt4\" (UID: \"8837b017-c001-4a7c-ae36-e582dd953e3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:16.855545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.855548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228sc\" (UniqueName: \"kubernetes.io/projected/8837b017-c001-4a7c-ae36-e582dd953e3a-kube-api-access-228sc\") pod \"cert-manager-cainjector-559cdd8c87-mcjt4\" (UID: \"8837b017-c001-4a7c-ae36-e582dd953e3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:16.956936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.956894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8837b017-c001-4a7c-ae36-e582dd953e3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mcjt4\" (UID: \"8837b017-c001-4a7c-ae36-e582dd953e3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:16.957123 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.956946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228sc\" (UniqueName: \"kubernetes.io/projected/8837b017-c001-4a7c-ae36-e582dd953e3a-kube-api-access-228sc\") pod \"cert-manager-cainjector-559cdd8c87-mcjt4\" (UID: \"8837b017-c001-4a7c-ae36-e582dd953e3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:16.966793 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.966760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8837b017-c001-4a7c-ae36-e582dd953e3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mcjt4\" (UID: \"8837b017-c001-4a7c-ae36-e582dd953e3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:16.966941 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:16.966804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-228sc\" (UniqueName: \"kubernetes.io/projected/8837b017-c001-4a7c-ae36-e582dd953e3a-kube-api-access-228sc\") pod \"cert-manager-cainjector-559cdd8c87-mcjt4\" (UID: \"8837b017-c001-4a7c-ae36-e582dd953e3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:17.091072 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:17.090979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" Jul 22 16:27:17.831784 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:17.831755 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4"] Jul 22 16:27:17.833990 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:27:17.833963 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8837b017_c001_4a7c_ae36_e582dd953e3a.slice/crio-fd7daaf84accef95520b6d34e874d2e95c853db62de81f766c3addd76209126d WatchSource:0}: Error finding container fd7daaf84accef95520b6d34e874d2e95c853db62de81f766c3addd76209126d: Status 404 returned error can't find the container with id fd7daaf84accef95520b6d34e874d2e95c853db62de81f766c3addd76209126d Jul 22 16:27:18.694309 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:18.694272 2580 generic.go:356] "Generic (PLEG): container finished" podID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerID="6cd1ec189c972ba021a52a6dd9ede3119864b7e810b3b75d918e114a38c0b55a" exitCode=0 Jul 22 16:27:18.694572 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:18.694347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" event={"ID":"01072ed1-780d-46ba-a1ae-e10c135c6551","Type":"ContainerDied","Data":"6cd1ec189c972ba021a52a6dd9ede3119864b7e810b3b75d918e114a38c0b55a"} Jul 22 16:27:18.695816 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:18.695784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" event={"ID":"8837b017-c001-4a7c-ae36-e582dd953e3a","Type":"ContainerStarted","Data":"fd7daaf84accef95520b6d34e874d2e95c853db62de81f766c3addd76209126d"} Jul 22 16:27:19.703556 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:19.703524 2580 generic.go:356] "Generic (PLEG): container finished" podID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerID="356ff6a150b1ce04d671c486082f0c10499518133b6ee8d919bc1fa54f331462" exitCode=0 Jul 22 16:27:19.703983 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:19.703603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" event={"ID":"01072ed1-780d-46ba-a1ae-e10c135c6551","Type":"ContainerDied","Data":"356ff6a150b1ce04d671c486082f0c10499518133b6ee8d919bc1fa54f331462"} Jul 22 16:27:20.710363 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.710325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" event={"ID":"8837b017-c001-4a7c-ae36-e582dd953e3a","Type":"ContainerStarted","Data":"1f7080c6d3f1245ea951f241a39eb5b52b79708a9d7bbedabf38b1aeef2f484e"} Jul 22 16:27:20.739053 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.738999 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mcjt4" podStartSLOduration=2.108171113 podStartE2EDuration="4.738982046s" totalImagesPullingTime="2.630810933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:27:20.736939162 +0000 UTC m=+398.317033077" watchObservedRunningTime="2026-07-22 16:27:20.738982046 +0000 UTC m=+398.319075944" Jul 22 16:27:20.857735 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.857705 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:20.996679 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.996604 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9s6q\" (UniqueName: \"kubernetes.io/projected/01072ed1-780d-46ba-a1ae-e10c135c6551-kube-api-access-p9s6q\") pod \"01072ed1-780d-46ba-a1ae-e10c135c6551\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " Jul 22 16:27:20.996679 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.996655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-bundle\") pod \"01072ed1-780d-46ba-a1ae-e10c135c6551\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " Jul 22 16:27:20.996864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.996686 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-util\") pod \"01072ed1-780d-46ba-a1ae-e10c135c6551\" (UID: \"01072ed1-780d-46ba-a1ae-e10c135c6551\") " Jul 22 16:27:20.997067 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.997039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-bundle" (OuterVolumeSpecName: "bundle") pod "01072ed1-780d-46ba-a1ae-e10c135c6551" (UID: "01072ed1-780d-46ba-a1ae-e10c135c6551"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:20.998795 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:20.998772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01072ed1-780d-46ba-a1ae-e10c135c6551-kube-api-access-p9s6q" (OuterVolumeSpecName: "kube-api-access-p9s6q") pod "01072ed1-780d-46ba-a1ae-e10c135c6551" (UID: "01072ed1-780d-46ba-a1ae-e10c135c6551"). InnerVolumeSpecName "kube-api-access-p9s6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:27:21.001192 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.001151 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-util" (OuterVolumeSpecName: "util") pod "01072ed1-780d-46ba-a1ae-e10c135c6551" (UID: "01072ed1-780d-46ba-a1ae-e10c135c6551"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:21.097581 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.097548 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s6q\" (UniqueName: \"kubernetes.io/projected/01072ed1-780d-46ba-a1ae-e10c135c6551-kube-api-access-p9s6q\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:21.097581 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.097575 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:21.097581 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.097584 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01072ed1-780d-46ba-a1ae-e10c135c6551-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:21.716993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.716964 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" Jul 22 16:27:21.717345 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.716993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff8ddh" event={"ID":"01072ed1-780d-46ba-a1ae-e10c135c6551","Type":"ContainerDied","Data":"43d7749f793f524799140c578e44223c0284cc4bf15029f025ef3d71251308b7"} Jul 22 16:27:21.717345 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:21.717039 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d7749f793f524799140c578e44223c0284cc4bf15029f025ef3d71251308b7" Jul 22 16:27:25.659462 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659422 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-77b8c"] Jul 22 16:27:25.659928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659889 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="pull" Jul 22 16:27:25.659928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659909 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="pull" Jul 22 16:27:25.659928 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659926 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="util" Jul 22 16:27:25.660090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659935 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="util" Jul 22 16:27:25.660090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659949 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="extract" Jul 22 16:27:25.660090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.659958 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="extract" Jul 22 16:27:25.660090 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.660080 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01072ed1-780d-46ba-a1ae-e10c135c6551" containerName="extract" Jul 22 16:27:25.664181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.664159 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.666320 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.666298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qg8c6\"" Jul 22 16:27:25.672597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.672575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-77b8c"] Jul 22 16:27:25.737352 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.737319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0cb2980-5e57-4df6-bfc2-2867c4d3e7af-bound-sa-token\") pod \"cert-manager-55b4d954d5-77b8c\" (UID: \"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af\") " pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.737509 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.737359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbkj\" (UniqueName: \"kubernetes.io/projected/f0cb2980-5e57-4df6-bfc2-2867c4d3e7af-kube-api-access-4qbkj\") pod \"cert-manager-55b4d954d5-77b8c\" (UID: \"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af\") " pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.837989 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.837955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0cb2980-5e57-4df6-bfc2-2867c4d3e7af-bound-sa-token\") pod \"cert-manager-55b4d954d5-77b8c\" (UID: \"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af\") " pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.838134 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.837997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbkj\" (UniqueName: \"kubernetes.io/projected/f0cb2980-5e57-4df6-bfc2-2867c4d3e7af-kube-api-access-4qbkj\") pod \"cert-manager-55b4d954d5-77b8c\" (UID: \"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af\") " pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.847241 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.847209 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0cb2980-5e57-4df6-bfc2-2867c4d3e7af-bound-sa-token\") pod \"cert-manager-55b4d954d5-77b8c\" (UID: \"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af\") " pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.847374 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.847354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbkj\" (UniqueName: \"kubernetes.io/projected/f0cb2980-5e57-4df6-bfc2-2867c4d3e7af-kube-api-access-4qbkj\") pod \"cert-manager-55b4d954d5-77b8c\" (UID: \"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af\") " pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:25.973838 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:25.973746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-77b8c" Jul 22 16:27:26.106244 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:26.106197 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-77b8c"] Jul 22 16:27:26.108470 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:27:26.108441 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0cb2980_5e57_4df6_bfc2_2867c4d3e7af.slice/crio-547c12d583190861aff8504b3f410114c02f41a70814ca60dab45b56f425a02e WatchSource:0}: Error finding container 547c12d583190861aff8504b3f410114c02f41a70814ca60dab45b56f425a02e: Status 404 returned error can't find the container with id 547c12d583190861aff8504b3f410114c02f41a70814ca60dab45b56f425a02e Jul 22 16:27:26.740463 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:26.740426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-77b8c" event={"ID":"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af","Type":"ContainerStarted","Data":"e7488cb6fbafe0ace957559a89d8b8242d89b9136bb8b46525f9df1c44d7cba6"} Jul 22 16:27:26.740463 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:26.740469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-77b8c" event={"ID":"f0cb2980-5e57-4df6-bfc2-2867c4d3e7af","Type":"ContainerStarted","Data":"547c12d583190861aff8504b3f410114c02f41a70814ca60dab45b56f425a02e"} Jul 22 16:27:26.758989 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:26.758932 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-77b8c" podStartSLOduration=1.758915587 podStartE2EDuration="1.758915587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:27:26.757766053 +0000 UTC m=+404.337859968" watchObservedRunningTime="2026-07-22 16:27:26.758915587 +0000 UTC m=+404.339009483" Jul 22 16:27:34.989885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:34.989842 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb"] Jul 22 16:27:34.995448 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:34.995426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:34.997861 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:34.997841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gj5ff\"" Jul 22 16:27:34.997979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:34.997926 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 16:27:34.997979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:34.997943 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 16:27:35.005594 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.004698 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb"] Jul 22 16:27:35.120126 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.120091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.120126 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.120130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.120432 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.120386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lbqt\" (UniqueName: \"kubernetes.io/projected/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-kube-api-access-4lbqt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.221269 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.221235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lbqt\" (UniqueName: \"kubernetes.io/projected/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-kube-api-access-4lbqt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.221493 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.221301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.221493 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.221321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.221643 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.221623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.221689 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.221654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.231537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.231517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lbqt\" (UniqueName: \"kubernetes.io/projected/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-kube-api-access-4lbqt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.310902 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.310844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:35.447863 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.447740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb"] Jul 22 16:27:35.450624 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:27:35.450595 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a647235_d9b1_4584_98cd_ee2dcd2a9a73.slice/crio-773d357d29885588f71afb9a2231b2dc5daaf5f70799a7788a8a968987bd573b WatchSource:0}: Error finding container 773d357d29885588f71afb9a2231b2dc5daaf5f70799a7788a8a968987bd573b: Status 404 returned error can't find the container with id 773d357d29885588f71afb9a2231b2dc5daaf5f70799a7788a8a968987bd573b Jul 22 16:27:35.783949 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.783909 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerID="035b43f4444bf2677d621d064a7a30800409bcd5c4022258d5cc02e1c4753466" exitCode=0 Jul 22 16:27:35.784129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.784000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" event={"ID":"6a647235-d9b1-4584-98cd-ee2dcd2a9a73","Type":"ContainerDied","Data":"035b43f4444bf2677d621d064a7a30800409bcd5c4022258d5cc02e1c4753466"} Jul 22 16:27:35.784129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:35.784050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" event={"ID":"6a647235-d9b1-4584-98cd-ee2dcd2a9a73","Type":"ContainerStarted","Data":"773d357d29885588f71afb9a2231b2dc5daaf5f70799a7788a8a968987bd573b"} Jul 22 16:27:36.790907 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:36.790869 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerID="18c32fbdf6f2de4bc05502773358d32ce43850d94cbbc5d97d4ecd74c0b70f48" exitCode=0 Jul 22 16:27:36.791285 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:36.790956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" event={"ID":"6a647235-d9b1-4584-98cd-ee2dcd2a9a73","Type":"ContainerDied","Data":"18c32fbdf6f2de4bc05502773358d32ce43850d94cbbc5d97d4ecd74c0b70f48"} Jul 22 16:27:37.796980 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:37.796948 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerID="9349b3469946c5cdbfa02ba312236fb0faf11fad34bf02d2f719f5d4c7761760" exitCode=0 Jul 22 16:27:37.797331 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:37.797025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" event={"ID":"6a647235-d9b1-4584-98cd-ee2dcd2a9a73","Type":"ContainerDied","Data":"9349b3469946c5cdbfa02ba312236fb0faf11fad34bf02d2f719f5d4c7761760"} Jul 22 16:27:38.928187 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:38.928162 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:39.054312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.054225 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-util\") pod \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " Jul 22 16:27:39.054312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.054279 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-bundle\") pod \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " Jul 22 16:27:39.054511 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.054433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lbqt\" (UniqueName: \"kubernetes.io/projected/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-kube-api-access-4lbqt\") pod \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\" (UID: \"6a647235-d9b1-4584-98cd-ee2dcd2a9a73\") " Jul 22 16:27:39.055118 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.055087 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-bundle" (OuterVolumeSpecName: "bundle") pod "6a647235-d9b1-4584-98cd-ee2dcd2a9a73" (UID: "6a647235-d9b1-4584-98cd-ee2dcd2a9a73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:39.056487 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.056462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-kube-api-access-4lbqt" (OuterVolumeSpecName: "kube-api-access-4lbqt") pod "6a647235-d9b1-4584-98cd-ee2dcd2a9a73" (UID: "6a647235-d9b1-4584-98cd-ee2dcd2a9a73"). InnerVolumeSpecName "kube-api-access-4lbqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:27:39.059925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.059900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-util" (OuterVolumeSpecName: "util") pod "6a647235-d9b1-4584-98cd-ee2dcd2a9a73" (UID: "6a647235-d9b1-4584-98cd-ee2dcd2a9a73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:39.155781 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.155748 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lbqt\" (UniqueName: \"kubernetes.io/projected/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-kube-api-access-4lbqt\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:39.155781 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.155775 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:39.155781 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.155786 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a647235-d9b1-4584-98cd-ee2dcd2a9a73-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:27:39.808943 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.808909 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" Jul 22 16:27:39.809123 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.808908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af892hb" event={"ID":"6a647235-d9b1-4584-98cd-ee2dcd2a9a73","Type":"ContainerDied","Data":"773d357d29885588f71afb9a2231b2dc5daaf5f70799a7788a8a968987bd573b"} Jul 22 16:27:39.809123 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:39.809031 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773d357d29885588f71afb9a2231b2dc5daaf5f70799a7788a8a968987bd573b" Jul 22 16:27:47.704570 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704530 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5"] Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704892 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="pull" Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704904 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="pull" Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704919 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="util" Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704924 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="util" Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704930 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="extract" Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704935 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="extract" Jul 22 16:27:47.705035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.704993 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a647235-d9b1-4584-98cd-ee2dcd2a9a73" containerName="extract" Jul 22 16:27:47.709410 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.709375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.712203 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.712174 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 16:27:47.712356 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.712208 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 16:27:47.712356 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.712250 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mxznl\"" Jul 22 16:27:47.712500 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.712351 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 16:27:47.712500 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.712363 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 16:27:47.721888 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.721849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5"] Jul 22 16:27:47.830283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.830247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-tmp\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.830283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.830286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.830534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.830327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.830534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.830368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtw6\" (UniqueName: \"kubernetes.io/projected/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-kube-api-access-tdtw6\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.830534 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.830438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-manifests\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.931761 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.931726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-tmp\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.931761 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.931762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.932014 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.931789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.932014 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.931808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtw6\" (UniqueName: \"kubernetes.io/projected/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-kube-api-access-tdtw6\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.932014 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.931831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-manifests\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.932286 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.932255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-manifests\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.934164 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.934134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-tmp\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.934308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.934293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.934378 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.934362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:47.948736 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:47.948713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtw6\" (UniqueName: \"kubernetes.io/projected/5b86b382-ed3c-4952-ad67-bcf3bc5c21a8-kube-api-access-tdtw6\") pod \"opendatahub-operator-controller-manager-6cb6494f8-hrnd5\" (UID: \"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:48.020235 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:48.020208 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:48.164607 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:48.164576 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5"] Jul 22 16:27:48.168512 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:27:48.168483 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b86b382_ed3c_4952_ad67_bcf3bc5c21a8.slice/crio-49818988c39359dd4700cba5da911dd23ed3460df0c707e166f79c0bfe96bea7 WatchSource:0}: Error finding container 49818988c39359dd4700cba5da911dd23ed3460df0c707e166f79c0bfe96bea7: Status 404 returned error can't find the container with id 49818988c39359dd4700cba5da911dd23ed3460df0c707e166f79c0bfe96bea7 Jul 22 16:27:48.852771 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:48.852691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" event={"ID":"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8","Type":"ContainerStarted","Data":"49818988c39359dd4700cba5da911dd23ed3460df0c707e166f79c0bfe96bea7"} Jul 22 16:27:51.876956 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:51.876919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" event={"ID":"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8","Type":"ContainerStarted","Data":"d4a6c31fcd16a0f80d937154f19af2f9dde964743866f96e7466671496a87bef"} Jul 22 16:27:52.882566 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:52.882534 2580 generic.go:356] "Generic (PLEG): container finished" podID="5b86b382-ed3c-4952-ad67-bcf3bc5c21a8" containerID="d4a6c31fcd16a0f80d937154f19af2f9dde964743866f96e7466671496a87bef" exitCode=0 Jul 22 16:27:52.882952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:52.882598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" event={"ID":"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8","Type":"ContainerDied","Data":"d4a6c31fcd16a0f80d937154f19af2f9dde964743866f96e7466671496a87bef"} Jul 22 16:27:53.890146 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:53.890111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" event={"ID":"5b86b382-ed3c-4952-ad67-bcf3bc5c21a8","Type":"ContainerStarted","Data":"596f81316fd62fa0d999bd2278fe139271dc5bc79e6b30a67f1dbd6f07a582a5"} Jul 22 16:27:53.890582 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:53.890352 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:27:53.915021 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:27:53.914975 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" podStartSLOduration=2.317664997 podStartE2EDuration="6.914959948s" totalImagesPullingTime="3.515889132s" totalInitContainerRuntime="1.081405819s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:27:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:27:53.91330979 +0000 UTC m=+431.493403686" watchObservedRunningTime="2026-07-22 16:27:53.914959948 +0000 UTC m=+431.495053844" Jul 22 16:28:04.897286 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:04.897255 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cb6494f8-hrnd5" Jul 22 16:28:07.485915 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.485806 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-788547964b-9bvkq"] Jul 22 16:28:07.489180 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.489161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.492418 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.492381 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 16:28:07.492642 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.492625 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 16:28:07.492728 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.492631 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-csw7k\"" Jul 22 16:28:07.509618 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.509588 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788547964b-9bvkq"] Jul 22 16:28:07.590818 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.590785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1813ab2b-baf9-423a-abd1-e3455c6a121c-tls-certs\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.590990 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.590859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfhh\" (UniqueName: \"kubernetes.io/projected/1813ab2b-baf9-423a-abd1-e3455c6a121c-kube-api-access-lxfhh\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.590990 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.590920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1813ab2b-baf9-423a-abd1-e3455c6a121c-tmp\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.678564 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.678525 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cxv2p"] Jul 22 16:28:07.681869 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.681852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.683925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.683895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 16:28:07.684042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.683927 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9rfwj\"" Jul 22 16:28:07.690830 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.690808 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cxv2p"] Jul 22 16:28:07.692176 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.692153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1813ab2b-baf9-423a-abd1-e3455c6a121c-tls-certs\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.692298 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.692267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfhh\" (UniqueName: \"kubernetes.io/projected/1813ab2b-baf9-423a-abd1-e3455c6a121c-kube-api-access-lxfhh\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.692364 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.692313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1813ab2b-baf9-423a-abd1-e3455c6a121c-tmp\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.694804 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.694780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1813ab2b-baf9-423a-abd1-e3455c6a121c-tmp\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.695030 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.695011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1813ab2b-baf9-423a-abd1-e3455c6a121c-tls-certs\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.706457 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.706428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfhh\" (UniqueName: \"kubernetes.io/projected/1813ab2b-baf9-423a-abd1-e3455c6a121c-kube-api-access-lxfhh\") pod \"kube-auth-proxy-788547964b-9bvkq\" (UID: \"1813ab2b-baf9-423a-abd1-e3455c6a121c\") " pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.793406 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.793354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/496a6e1c-f542-4f29-a10b-c5f93ac3b778-cert\") pod \"odh-model-controller-7998c67dc4-cxv2p\" (UID: \"496a6e1c-f542-4f29-a10b-c5f93ac3b778\") " pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.793606 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.793434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvzs\" (UniqueName: \"kubernetes.io/projected/496a6e1c-f542-4f29-a10b-c5f93ac3b778-kube-api-access-zsvzs\") pod \"odh-model-controller-7998c67dc4-cxv2p\" (UID: \"496a6e1c-f542-4f29-a10b-c5f93ac3b778\") " pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.799275 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.799252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" Jul 22 16:28:07.894455 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.894421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/496a6e1c-f542-4f29-a10b-c5f93ac3b778-cert\") pod \"odh-model-controller-7998c67dc4-cxv2p\" (UID: \"496a6e1c-f542-4f29-a10b-c5f93ac3b778\") " pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.894629 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.894500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvzs\" (UniqueName: \"kubernetes.io/projected/496a6e1c-f542-4f29-a10b-c5f93ac3b778-kube-api-access-zsvzs\") pod \"odh-model-controller-7998c67dc4-cxv2p\" (UID: \"496a6e1c-f542-4f29-a10b-c5f93ac3b778\") " pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.896894 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.896866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/496a6e1c-f542-4f29-a10b-c5f93ac3b778-cert\") pod \"odh-model-controller-7998c67dc4-cxv2p\" (UID: \"496a6e1c-f542-4f29-a10b-c5f93ac3b778\") " pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.905132 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.905105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvzs\" (UniqueName: \"kubernetes.io/projected/496a6e1c-f542-4f29-a10b-c5f93ac3b778-kube-api-access-zsvzs\") pod \"odh-model-controller-7998c67dc4-cxv2p\" (UID: \"496a6e1c-f542-4f29-a10b-c5f93ac3b778\") " pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:07.941313 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.941280 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788547964b-9bvkq"] Jul 22 16:28:07.944250 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:07.944218 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1813ab2b_baf9_423a_abd1_e3455c6a121c.slice/crio-74a2772d20554ad36dc77c480c98c28382127a61ab2e71cf05e5e6c901faea9d WatchSource:0}: Error finding container 74a2772d20554ad36dc77c480c98c28382127a61ab2e71cf05e5e6c901faea9d: Status 404 returned error can't find the container with id 74a2772d20554ad36dc77c480c98c28382127a61ab2e71cf05e5e6c901faea9d Jul 22 16:28:07.956731 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.956703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" event={"ID":"1813ab2b-baf9-423a-abd1-e3455c6a121c","Type":"ContainerStarted","Data":"74a2772d20554ad36dc77c480c98c28382127a61ab2e71cf05e5e6c901faea9d"} Jul 22 16:28:07.994602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:07.994562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:08.132755 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.132728 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cxv2p"] Jul 22 16:28:08.134489 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:08.134455 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496a6e1c_f542_4f29_a10b_c5f93ac3b778.slice/crio-83f3990370c6f330da2c3dfc19929a2ee3a9e48b9cc82ce03931c81939635eaf WatchSource:0}: Error finding container 83f3990370c6f330da2c3dfc19929a2ee3a9e48b9cc82ce03931c81939635eaf: Status 404 returned error can't find the container with id 83f3990370c6f330da2c3dfc19929a2ee3a9e48b9cc82ce03931c81939635eaf Jul 22 16:28:08.203426 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.203384 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8hgbt"] Jul 22 16:28:08.207821 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.207801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.210885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.210866 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 16:28:08.223740 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.223714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8hgbt"] Jul 22 16:28:08.298192 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.298093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba1aada-5cbf-4d18-a198-85165a7e2b1a-tls-certs\") pod \"model-serving-api-54d89bd5fc-8hgbt\" (UID: \"0ba1aada-5cbf-4d18-a198-85165a7e2b1a\") " pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.298192 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.298152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzx5v\" (UniqueName: \"kubernetes.io/projected/0ba1aada-5cbf-4d18-a198-85165a7e2b1a-kube-api-access-nzx5v\") pod \"model-serving-api-54d89bd5fc-8hgbt\" (UID: \"0ba1aada-5cbf-4d18-a198-85165a7e2b1a\") " pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.399080 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.399045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba1aada-5cbf-4d18-a198-85165a7e2b1a-tls-certs\") pod \"model-serving-api-54d89bd5fc-8hgbt\" (UID: \"0ba1aada-5cbf-4d18-a198-85165a7e2b1a\") " pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.399226 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.399101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzx5v\" (UniqueName: \"kubernetes.io/projected/0ba1aada-5cbf-4d18-a198-85165a7e2b1a-kube-api-access-nzx5v\") pod \"model-serving-api-54d89bd5fc-8hgbt\" (UID: \"0ba1aada-5cbf-4d18-a198-85165a7e2b1a\") " pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.401500 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.401480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba1aada-5cbf-4d18-a198-85165a7e2b1a-tls-certs\") pod \"model-serving-api-54d89bd5fc-8hgbt\" (UID: \"0ba1aada-5cbf-4d18-a198-85165a7e2b1a\") " pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.408695 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.408668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzx5v\" (UniqueName: \"kubernetes.io/projected/0ba1aada-5cbf-4d18-a198-85165a7e2b1a-kube-api-access-nzx5v\") pod \"model-serving-api-54d89bd5fc-8hgbt\" (UID: \"0ba1aada-5cbf-4d18-a198-85165a7e2b1a\") " pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.518866 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.518819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:08.708185 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.708151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8hgbt"] Jul 22 16:28:08.711436 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:08.711371 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba1aada_5cbf_4d18_a198_85165a7e2b1a.slice/crio-20dc96e3b4a8a27fef91bddc6fba8407cebfab06e95c75227f90b9140866ab2a WatchSource:0}: Error finding container 20dc96e3b4a8a27fef91bddc6fba8407cebfab06e95c75227f90b9140866ab2a: Status 404 returned error can't find the container with id 20dc96e3b4a8a27fef91bddc6fba8407cebfab06e95c75227f90b9140866ab2a Jul 22 16:28:08.963936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.963843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" event={"ID":"0ba1aada-5cbf-4d18-a198-85165a7e2b1a","Type":"ContainerStarted","Data":"20dc96e3b4a8a27fef91bddc6fba8407cebfab06e95c75227f90b9140866ab2a"} Jul 22 16:28:08.964960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.964933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" event={"ID":"496a6e1c-f542-4f29-a10b-c5f93ac3b778","Type":"ContainerStarted","Data":"83f3990370c6f330da2c3dfc19929a2ee3a9e48b9cc82ce03931c81939635eaf"} Jul 22 16:28:08.984256 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.984222 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n2222"] Jul 22 16:28:08.988477 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.988446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:08.993559 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.993529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 16:28:08.993709 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.993531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 16:28:08.994135 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.993968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8m4ks\"" Jul 22 16:28:08.994135 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.994019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 16:28:08.994135 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:08.994031 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 16:28:09.003099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.003064 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n2222"] Jul 22 16:28:09.106195 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.106409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrhz\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-kube-api-access-czrhz\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.106409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.106409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/568c176d-1ab2-4282-8b14-158ed1ba52d1-local-certs\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.106409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-token\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.106635 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.106635 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.106462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-cacerts\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207559 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/568c176d-1ab2-4282-8b14-158ed1ba52d1-local-certs\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207749 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-token\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207749 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207749 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-cacerts\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207749 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrhz\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-kube-api-access-czrhz\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.207952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.207783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.208525 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.208480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.211319 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.211290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.211457 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.211420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/568c176d-1ab2-4282-8b14-158ed1ba52d1-local-certs\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.211925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.211797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-cacerts\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.212646 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.212606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.218308 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.218226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrhz\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-kube-api-access-czrhz\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.218618 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.218596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-token\") pod \"istiod-openshift-gateway-7574459656-n2222\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.304774 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.304737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:09.519445 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.518311 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n2222"] Jul 22 16:28:09.523659 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:09.523624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568c176d_1ab2_4282_8b14_158ed1ba52d1.slice/crio-8a181ae81d23b393931e7322ea3957333be87073b225a4eee049fb17fd21174d WatchSource:0}: Error finding container 8a181ae81d23b393931e7322ea3957333be87073b225a4eee049fb17fd21174d: Status 404 returned error can't find the container with id 8a181ae81d23b393931e7322ea3957333be87073b225a4eee049fb17fd21174d Jul 22 16:28:09.973649 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:09.973557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" event={"ID":"568c176d-1ab2-4282-8b14-158ed1ba52d1","Type":"ContainerStarted","Data":"8a181ae81d23b393931e7322ea3957333be87073b225a4eee049fb17fd21174d"} Jul 22 16:28:12.994312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:12.994269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" event={"ID":"1813ab2b-baf9-423a-abd1-e3455c6a121c","Type":"ContainerStarted","Data":"89158ec8757e2a455ddd4d1ba505b389270800a160e8c7188fa971c513896bac"} Jul 22 16:28:12.996088 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:12.996058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" event={"ID":"496a6e1c-f542-4f29-a10b-c5f93ac3b778","Type":"ContainerStarted","Data":"695a0c86f15eac9a9f37f7537df8a6e0072c5d319bda853477945e3a53cb71fd"} Jul 22 16:28:12.996231 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:12.996137 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:12.997730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:12.997701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" event={"ID":"0ba1aada-5cbf-4d18-a198-85165a7e2b1a","Type":"ContainerStarted","Data":"07612600011fe0ad05e40e5293d09b3920247c47c3e8d03e6d563681c49fe36c"} Jul 22 16:28:12.997852 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:12.997773 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:13.020333 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:13.020279 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-788547964b-9bvkq" podStartSLOduration=1.770150805 podStartE2EDuration="6.020256499s" totalImagesPullingTime="4.250105694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:13.019759455 +0000 UTC m=+450.599853357" watchObservedRunningTime="2026-07-22 16:28:13.020256499 +0000 UTC m=+450.600350396" Jul 22 16:28:13.059388 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:13.059325 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" podStartSLOduration=2.005991028 podStartE2EDuration="6.059301829s" totalImagesPullingTime="4.053310801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:13.058389891 +0000 UTC m=+450.638483790" watchObservedRunningTime="2026-07-22 16:28:13.059301829 +0000 UTC m=+450.639395726" Jul 22 16:28:13.095602 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:13.095192 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" podStartSLOduration=1.620143562 podStartE2EDuration="5.095167321s" totalImagesPullingTime="3.475023759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:13.085632437 +0000 UTC m=+450.665726331" watchObservedRunningTime="2026-07-22 16:28:13.095167321 +0000 UTC m=+450.675261218" Jul 22 16:28:13.496455 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:13.496422 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:28:13.496537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:13.496485 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:28:14.004023 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:14.003985 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" event={"ID":"568c176d-1ab2-4282-8b14-158ed1ba52d1","Type":"ContainerStarted","Data":"99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0"} Jul 22 16:28:14.004465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:14.004213 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:14.006127 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:14.006097 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-n2222 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 16:28:14.006236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:14.006160 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:28:14.031597 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:14.031553 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" podStartSLOduration=2.06161621 podStartE2EDuration="6.031538331s" totalImagesPullingTime="3.969922121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:14.026425699 +0000 UTC m=+451.606519597" watchObservedRunningTime="2026-07-22 16:28:14.031538331 +0000 UTC m=+451.611632227" Jul 22 16:28:15.009757 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:15.009727 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-n2222 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 16:28:15.010135 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:15.009773 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:28:18.009925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.009894 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:18.389544 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.389436 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p"] Jul 22 16:28:18.393256 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.393234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.395828 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.395809 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-k8j48\"" Jul 22 16:28:18.396043 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.396024 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 16:28:18.403028 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.403005 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f"] Jul 22 16:28:18.406545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.406524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:18.407472 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.407448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p"] Jul 22 16:28:18.408888 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.408863 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 16:28:18.409367 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.409026 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fh7hq\"" Jul 22 16:28:18.422713 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.422691 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f"] Jul 22 16:28:18.493959 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.493924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sfhk\" (UniqueName: \"kubernetes.io/projected/223fb4dd-7b0e-4180-858c-7dd45bb35055-kube-api-access-8sfhk\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:18.494114 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.493966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/223fb4dd-7b0e-4180-858c-7dd45bb35055-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:18.494114 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.494023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e8e5f48-3262-4af7-9282-035d8da49edd-cert\") pod \"kserve-controller-manager-d6f4b65b8-w6d8p\" (UID: \"1e8e5f48-3262-4af7-9282-035d8da49edd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.494114 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.494047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqng\" (UniqueName: \"kubernetes.io/projected/1e8e5f48-3262-4af7-9282-035d8da49edd-kube-api-access-gxqng\") pod \"kserve-controller-manager-d6f4b65b8-w6d8p\" (UID: \"1e8e5f48-3262-4af7-9282-035d8da49edd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.595548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.595511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sfhk\" (UniqueName: \"kubernetes.io/projected/223fb4dd-7b0e-4180-858c-7dd45bb35055-kube-api-access-8sfhk\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:18.595725 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.595555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/223fb4dd-7b0e-4180-858c-7dd45bb35055-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:18.595725 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.595606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e8e5f48-3262-4af7-9282-035d8da49edd-cert\") pod \"kserve-controller-manager-d6f4b65b8-w6d8p\" (UID: \"1e8e5f48-3262-4af7-9282-035d8da49edd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.595725 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.595634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqng\" (UniqueName: \"kubernetes.io/projected/1e8e5f48-3262-4af7-9282-035d8da49edd-kube-api-access-gxqng\") pod \"kserve-controller-manager-d6f4b65b8-w6d8p\" (UID: \"1e8e5f48-3262-4af7-9282-035d8da49edd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.595919 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:28:18.595722 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 22 16:28:18.595919 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:28:18.595779 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223fb4dd-7b0e-4180-858c-7dd45bb35055-cert podName:223fb4dd-7b0e-4180-858c-7dd45bb35055 nodeName:}" failed. No retries permitted until 2026-07-22 16:28:19.095763908 +0000 UTC m=+456.675857789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/223fb4dd-7b0e-4180-858c-7dd45bb35055-cert") pod "llmisvc-controller-manager-777c6bd7d5-7bf2f" (UID: "223fb4dd-7b0e-4180-858c-7dd45bb35055") : secret "llmisvc-webhook-server-cert" not found Jul 22 16:28:18.598051 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.598024 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e8e5f48-3262-4af7-9282-035d8da49edd-cert\") pod \"kserve-controller-manager-d6f4b65b8-w6d8p\" (UID: \"1e8e5f48-3262-4af7-9282-035d8da49edd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.607669 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.607646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sfhk\" (UniqueName: \"kubernetes.io/projected/223fb4dd-7b0e-4180-858c-7dd45bb35055-kube-api-access-8sfhk\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:18.607842 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.607821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqng\" (UniqueName: \"kubernetes.io/projected/1e8e5f48-3262-4af7-9282-035d8da49edd-kube-api-access-gxqng\") pod \"kserve-controller-manager-d6f4b65b8-w6d8p\" (UID: \"1e8e5f48-3262-4af7-9282-035d8da49edd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.703712 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.703623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:18.838849 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:18.838817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p"] Jul 22 16:28:18.842090 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:18.842063 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8e5f48_3262_4af7_9282_035d8da49edd.slice/crio-cbb7529640c916786e6e65fb7e60855eca18030b410afd1f3ca281f0180cec69 WatchSource:0}: Error finding container cbb7529640c916786e6e65fb7e60855eca18030b410afd1f3ca281f0180cec69: Status 404 returned error can't find the container with id cbb7529640c916786e6e65fb7e60855eca18030b410afd1f3ca281f0180cec69 Jul 22 16:28:19.028133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:19.028100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" event={"ID":"1e8e5f48-3262-4af7-9282-035d8da49edd","Type":"ContainerStarted","Data":"cbb7529640c916786e6e65fb7e60855eca18030b410afd1f3ca281f0180cec69"} Jul 22 16:28:19.100707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:19.100677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/223fb4dd-7b0e-4180-858c-7dd45bb35055-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:19.103066 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:19.103045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/223fb4dd-7b0e-4180-858c-7dd45bb35055-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bf2f\" (UID: \"223fb4dd-7b0e-4180-858c-7dd45bb35055\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:19.316170 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:19.316090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:19.456803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:19.456632 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f"] Jul 22 16:28:19.458959 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:19.458929 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod223fb4dd_7b0e_4180_858c_7dd45bb35055.slice/crio-efa8ba53878c77873ecab7151a511d0e86bed7c0a0dba37b1b14ebc854ca61cd WatchSource:0}: Error finding container efa8ba53878c77873ecab7151a511d0e86bed7c0a0dba37b1b14ebc854ca61cd: Status 404 returned error can't find the container with id efa8ba53878c77873ecab7151a511d0e86bed7c0a0dba37b1b14ebc854ca61cd Jul 22 16:28:20.035272 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:20.035242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" event={"ID":"223fb4dd-7b0e-4180-858c-7dd45bb35055","Type":"ContainerStarted","Data":"efa8ba53878c77873ecab7151a511d0e86bed7c0a0dba37b1b14ebc854ca61cd"} Jul 22 16:28:20.984126 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:20.984091 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n2222"] Jul 22 16:28:20.984351 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:20.984304 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" containerID="cri-o://99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0" gracePeriod=30 Jul 22 16:28:21.009604 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.009576 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-n2222 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:8080/ready\": dial tcp 10.133.0.35:8080: connect: connection refused" start-of-body= Jul 22 16:28:21.009742 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.009632 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.35:8080/ready\": dial tcp 10.133.0.35:8080: connect: connection refused" Jul 22 16:28:21.765975 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.765951 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:21.824584 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824550 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-kubeconfig\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.824765 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824616 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-token\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.824765 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824639 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czrhz\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-kube-api-access-czrhz\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.824765 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-ca-configmap\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.824931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824836 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/568c176d-1ab2-4282-8b14-158ed1ba52d1-local-certs\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.824931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-cacerts\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.825025 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.824927 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-dns-cert\") pod \"568c176d-1ab2-4282-8b14-158ed1ba52d1\" (UID: \"568c176d-1ab2-4282-8b14-158ed1ba52d1\") " Jul 22 16:28:21.825516 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.825091 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:28:21.825516 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.825258 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-ca-configmap\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:21.827590 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.827554 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568c176d-1ab2-4282-8b14-158ed1ba52d1-local-certs" (OuterVolumeSpecName: "local-certs") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:28:21.827890 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.827840 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-kube-api-access-czrhz" (OuterVolumeSpecName: "kube-api-access-czrhz") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "kube-api-access-czrhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:28:21.827890 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.827859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:28:21.827890 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.827859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:28:21.828603 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.828571 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:28:21.828794 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.828758 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-cacerts" (OuterVolumeSpecName: "cacerts") pod "568c176d-1ab2-4282-8b14-158ed1ba52d1" (UID: "568c176d-1ab2-4282-8b14-158ed1ba52d1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:28:21.926360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.926280 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-cacerts\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:21.926360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.926313 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-csr-dns-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:21.926360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.926324 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-kubeconfig\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:21.926360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.926333 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-istio-token\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:21.926360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.926343 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czrhz\" (UniqueName: \"kubernetes.io/projected/568c176d-1ab2-4282-8b14-158ed1ba52d1-kube-api-access-czrhz\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:21.926360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:21.926355 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/568c176d-1ab2-4282-8b14-158ed1ba52d1-local-certs\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:28:22.048989 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.048950 2580 generic.go:356] "Generic (PLEG): container finished" podID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerID="99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0" exitCode=0 Jul 22 16:28:22.049161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.049006 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" Jul 22 16:28:22.049161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.049039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" event={"ID":"568c176d-1ab2-4282-8b14-158ed1ba52d1","Type":"ContainerDied","Data":"99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0"} Jul 22 16:28:22.049161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.049097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n2222" event={"ID":"568c176d-1ab2-4282-8b14-158ed1ba52d1","Type":"ContainerDied","Data":"8a181ae81d23b393931e7322ea3957333be87073b225a4eee049fb17fd21174d"} Jul 22 16:28:22.049161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.049124 2580 scope.go:117] "RemoveContainer" containerID="99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0" Jul 22 16:28:22.051240 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.051212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" event={"ID":"1e8e5f48-3262-4af7-9282-035d8da49edd","Type":"ContainerStarted","Data":"225918434d4ca049fb75bd4d67334805ab8fec641211e05b0a662cf50699ef6a"} Jul 22 16:28:22.051369 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.051353 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:22.075563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.075518 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" podStartSLOduration=1.250190662 podStartE2EDuration="4.075502205s" totalImagesPullingTime="2.825311543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:22.073180842 +0000 UTC m=+459.653274758" watchObservedRunningTime="2026-07-22 16:28:22.075502205 +0000 UTC m=+459.655596109" Jul 22 16:28:22.093309 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.093277 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n2222"] Jul 22 16:28:22.105298 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.105271 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n2222"] Jul 22 16:28:22.403969 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.403947 2580 scope.go:117] "RemoveContainer" containerID="99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0" Jul 22 16:28:22.404257 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:28:22.404241 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0\": container with ID starting with 99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0 not found: ID does not exist" containerID="99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0" Jul 22 16:28:22.404293 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.404275 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0"} err="failed to get container status \"99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0\": rpc error: code = NotFound desc = could not find container \"99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0\": container with ID starting with 99d14cce27f7c16c31f574588be7716232f3bae3585105d932f01e86c97cdbc0 not found: ID does not exist" Jul 22 16:28:22.980407 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:22.980360 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" path="/var/lib/kubelet/pods/568c176d-1ab2-4282-8b14-158ed1ba52d1/volumes" Jul 22 16:28:23.057981 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:23.057939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" event={"ID":"223fb4dd-7b0e-4180-858c-7dd45bb35055","Type":"ContainerStarted","Data":"44edf49cd7fb7e8ce858f48eb6e6f32ab6a67084c4410c3e1916c8202810296a"} Jul 22 16:28:23.058177 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:23.058113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:23.086784 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:23.086734 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" podStartSLOduration=2.094092088 podStartE2EDuration="5.086715022s" totalImagesPullingTime="2.992622934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:23.084227149 +0000 UTC m=+460.664321068" watchObservedRunningTime="2026-07-22 16:28:23.086715022 +0000 UTC m=+460.666808920" Jul 22 16:28:24.007362 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:24.007332 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cxv2p" Jul 22 16:28:24.009474 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:24.009450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8hgbt" Jul 22 16:28:43.961737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:43.961699 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-755ddcb497-6kgl5"] Jul 22 16:28:43.962122 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:43.962080 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" Jul 22 16:28:43.962122 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:43.962092 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" Jul 22 16:28:43.962196 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:43.962180 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="568c176d-1ab2-4282-8b14-158ed1ba52d1" containerName="discovery" Jul 22 16:28:43.970117 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:43.970093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:43.979469 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:43.979442 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755ddcb497-6kgl5"] Jul 22 16:28:44.028799 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.028765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-trusted-ca-bundle\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.028799 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.028810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-oauth-config\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.029056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.028842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-oauth-serving-cert\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.029056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.028929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-serving-cert\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.029056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.028967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-config\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.029056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.029007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-service-ca\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.029056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.029043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfhm\" (UniqueName: \"kubernetes.io/projected/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-kube-api-access-7cfhm\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.129779 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-serving-cert\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.129971 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-config\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.129971 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-service-ca\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.129971 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfhm\" (UniqueName: \"kubernetes.io/projected/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-kube-api-access-7cfhm\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.129971 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-trusted-ca-bundle\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.129971 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-oauth-config\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.130229 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.129976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-oauth-serving-cert\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.130671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.130642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-config\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.130806 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.130724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-oauth-serving-cert\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.130806 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.130771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-service-ca\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.130975 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.130951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-trusted-ca-bundle\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.132337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.132317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-serving-cert\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.132454 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.132436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-console-oauth-config\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.139117 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.139095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfhm\" (UniqueName: \"kubernetes.io/projected/b258aea6-8d3c-4adf-ab09-27af9e8f2c18-kube-api-access-7cfhm\") pod \"console-755ddcb497-6kgl5\" (UID: \"b258aea6-8d3c-4adf-ab09-27af9e8f2c18\") " pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.280740 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.280712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:44.442977 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:44.442952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755ddcb497-6kgl5"] Jul 22 16:28:44.445259 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:28:44.445227 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb258aea6_8d3c_4adf_ab09_27af9e8f2c18.slice/crio-5cb066061c29eba0e2321b89b1d180a4ebc5b077a906c32c01d450dd89e1659a WatchSource:0}: Error finding container 5cb066061c29eba0e2321b89b1d180a4ebc5b077a906c32c01d450dd89e1659a: Status 404 returned error can't find the container with id 5cb066061c29eba0e2321b89b1d180a4ebc5b077a906c32c01d450dd89e1659a Jul 22 16:28:45.178725 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:45.178691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755ddcb497-6kgl5" event={"ID":"b258aea6-8d3c-4adf-ab09-27af9e8f2c18","Type":"ContainerStarted","Data":"94d0877219032f1b4a1d94449ac6c7ead89e98ce981be55cca5f02925b9200a6"} Jul 22 16:28:45.179206 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:45.178745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755ddcb497-6kgl5" event={"ID":"b258aea6-8d3c-4adf-ab09-27af9e8f2c18","Type":"ContainerStarted","Data":"5cb066061c29eba0e2321b89b1d180a4ebc5b077a906c32c01d450dd89e1659a"} Jul 22 16:28:45.198240 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:45.198184 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-755ddcb497-6kgl5" podStartSLOduration=2.198168804 podStartE2EDuration="2.198168804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:28:45.195856012 +0000 UTC m=+482.775949910" watchObservedRunningTime="2026-07-22 16:28:45.198168804 +0000 UTC m=+482.778262701" Jul 22 16:28:53.064693 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:53.064661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w6d8p" Jul 22 16:28:54.067207 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:54.067174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bf2f" Jul 22 16:28:54.281945 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:54.281904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:54.282125 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:54.281975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:54.286990 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:54.286963 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:55.229713 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:55.229679 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-755ddcb497-6kgl5" Jul 22 16:28:55.277916 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:28:55.277885 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59685fb89b-vlj8s"] Jul 22 16:29:08.963992 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:08.963955 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm486"] Jul 22 16:29:08.967837 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:08.967810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:08.970140 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:08.970107 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 16:29:08.970279 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:08.970106 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sxgjb\"" Jul 22 16:29:08.970854 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:08.970836 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 16:29:08.982364 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:08.982336 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm486"] Jul 22 16:29:09.069837 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.069787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxh8\" (UniqueName: \"kubernetes.io/projected/0855483d-f380-4a59-8c37-4925781eeda3-kube-api-access-sjxh8\") pod \"kuadrant-operator-catalog-gm486\" (UID: \"0855483d-f380-4a59-8c37-4925781eeda3\") " pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:09.170332 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.170298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxh8\" (UniqueName: \"kubernetes.io/projected/0855483d-f380-4a59-8c37-4925781eeda3-kube-api-access-sjxh8\") pod \"kuadrant-operator-catalog-gm486\" (UID: \"0855483d-f380-4a59-8c37-4925781eeda3\") " pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:09.179249 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.179223 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxh8\" (UniqueName: \"kubernetes.io/projected/0855483d-f380-4a59-8c37-4925781eeda3-kube-api-access-sjxh8\") pod \"kuadrant-operator-catalog-gm486\" (UID: \"0855483d-f380-4a59-8c37-4925781eeda3\") " pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:09.280719 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.280670 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:09.316976 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.316938 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm486"] Jul 22 16:29:09.419906 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.419875 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm486"] Jul 22 16:29:09.422047 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:09.422021 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0855483d_f380_4a59_8c37_4925781eeda3.slice/crio-771a7c2a1b7979fd3d344779638ee9202e91317d419f5d364298ea35a7e59a73 WatchSource:0}: Error finding container 771a7c2a1b7979fd3d344779638ee9202e91317d419f5d364298ea35a7e59a73: Status 404 returned error can't find the container with id 771a7c2a1b7979fd3d344779638ee9202e91317d419f5d364298ea35a7e59a73 Jul 22 16:29:09.525948 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.525908 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hm6sf"] Jul 22 16:29:09.530664 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.530647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:09.536895 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.536828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hm6sf"] Jul 22 16:29:09.675525 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.675474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp245\" (UniqueName: \"kubernetes.io/projected/79f7597d-f459-4da9-a7c3-9c8da9a82661-kube-api-access-vp245\") pod \"kuadrant-operator-catalog-hm6sf\" (UID: \"79f7597d-f459-4da9-a7c3-9c8da9a82661\") " pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:09.776323 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.776282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp245\" (UniqueName: \"kubernetes.io/projected/79f7597d-f459-4da9-a7c3-9c8da9a82661-kube-api-access-vp245\") pod \"kuadrant-operator-catalog-hm6sf\" (UID: \"79f7597d-f459-4da9-a7c3-9c8da9a82661\") " pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:09.785549 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.785515 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp245\" (UniqueName: \"kubernetes.io/projected/79f7597d-f459-4da9-a7c3-9c8da9a82661-kube-api-access-vp245\") pod \"kuadrant-operator-catalog-hm6sf\" (UID: \"79f7597d-f459-4da9-a7c3-9c8da9a82661\") " pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:09.842832 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.842748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:09.993040 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:09.993015 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hm6sf"] Jul 22 16:29:10.027878 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:10.027846 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f7597d_f459_4da9_a7c3_9c8da9a82661.slice/crio-9d1ca19d530b8f3824dac520aae0ffc3e188d447d0d95e9104e12f5c9d7cc58e WatchSource:0}: Error finding container 9d1ca19d530b8f3824dac520aae0ffc3e188d447d0d95e9104e12f5c9d7cc58e: Status 404 returned error can't find the container with id 9d1ca19d530b8f3824dac520aae0ffc3e188d447d0d95e9104e12f5c9d7cc58e Jul 22 16:29:10.304414 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:10.304355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm486" event={"ID":"0855483d-f380-4a59-8c37-4925781eeda3","Type":"ContainerStarted","Data":"771a7c2a1b7979fd3d344779638ee9202e91317d419f5d364298ea35a7e59a73"} Jul 22 16:29:10.306006 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:10.305967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" event={"ID":"79f7597d-f459-4da9-a7c3-9c8da9a82661","Type":"ContainerStarted","Data":"9d1ca19d530b8f3824dac520aae0ffc3e188d447d0d95e9104e12f5c9d7cc58e"} Jul 22 16:29:12.319546 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.319505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" event={"ID":"79f7597d-f459-4da9-a7c3-9c8da9a82661","Type":"ContainerStarted","Data":"e2835380b8606ddfb1686b31460d8cd130e471e66bf520571048676d03523067"} Jul 22 16:29:12.321048 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.321025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm486" event={"ID":"0855483d-f380-4a59-8c37-4925781eeda3","Type":"ContainerStarted","Data":"051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a"} Jul 22 16:29:12.321154 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.321119 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gm486" podUID="0855483d-f380-4a59-8c37-4925781eeda3" containerName="registry-server" containerID="cri-o://051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a" gracePeriod=2 Jul 22 16:29:12.336890 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.336840 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" podStartSLOduration=1.90585061 podStartE2EDuration="3.336823246s" totalImagesPullingTime="1.430972636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:12.335492123 +0000 UTC m=+509.915586045" watchObservedRunningTime="2026-07-22 16:29:12.336823246 +0000 UTC m=+509.916917154" Jul 22 16:29:12.351643 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.351598 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gm486" podStartSLOduration=2.317855327 podStartE2EDuration="4.351582899s" totalImagesPullingTime="2.033727572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:12.350425756 +0000 UTC m=+509.930519653" watchObservedRunningTime="2026-07-22 16:29:12.351582899 +0000 UTC m=+509.931676795" Jul 22 16:29:12.565356 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.565329 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:12.706015 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.705907 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjxh8\" (UniqueName: \"kubernetes.io/projected/0855483d-f380-4a59-8c37-4925781eeda3-kube-api-access-sjxh8\") pod \"0855483d-f380-4a59-8c37-4925781eeda3\" (UID: \"0855483d-f380-4a59-8c37-4925781eeda3\") " Jul 22 16:29:12.708088 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.708063 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0855483d-f380-4a59-8c37-4925781eeda3-kube-api-access-sjxh8" (OuterVolumeSpecName: "kube-api-access-sjxh8") pod "0855483d-f380-4a59-8c37-4925781eeda3" (UID: "0855483d-f380-4a59-8c37-4925781eeda3"). InnerVolumeSpecName "kube-api-access-sjxh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:12.807222 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:12.807174 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjxh8\" (UniqueName: \"kubernetes.io/projected/0855483d-f380-4a59-8c37-4925781eeda3-kube-api-access-sjxh8\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:13.327686 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.327590 2580 generic.go:356] "Generic (PLEG): container finished" podID="0855483d-f380-4a59-8c37-4925781eeda3" containerID="051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a" exitCode=0 Jul 22 16:29:13.327686 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.327649 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm486" Jul 22 16:29:13.328184 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.327680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm486" event={"ID":"0855483d-f380-4a59-8c37-4925781eeda3","Type":"ContainerDied","Data":"051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a"} Jul 22 16:29:13.328184 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.327739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm486" event={"ID":"0855483d-f380-4a59-8c37-4925781eeda3","Type":"ContainerDied","Data":"771a7c2a1b7979fd3d344779638ee9202e91317d419f5d364298ea35a7e59a73"} Jul 22 16:29:13.328184 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.327768 2580 scope.go:117] "RemoveContainer" containerID="051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a" Jul 22 16:29:13.337327 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.337307 2580 scope.go:117] "RemoveContainer" containerID="051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a" Jul 22 16:29:13.337592 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:13.337573 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a\": container with ID starting with 051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a not found: ID does not exist" containerID="051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a" Jul 22 16:29:13.337641 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.337600 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a"} err="failed to get container status \"051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a\": rpc error: code = NotFound desc = could not find container \"051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a\": container with ID starting with 051076f2f850e907f8c00e2d6beab2c5e44bededf091acaa4f6a41a667d5895a not found: ID does not exist" Jul 22 16:29:13.346306 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.346276 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm486"] Jul 22 16:29:13.350178 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:13.350157 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm486"] Jul 22 16:29:14.979958 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:14.979924 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0855483d-f380-4a59-8c37-4925781eeda3" path="/var/lib/kubelet/pods/0855483d-f380-4a59-8c37-4925781eeda3/volumes" Jul 22 16:29:19.843225 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:19.843184 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:19.843715 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:19.843258 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:19.866406 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:19.866366 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:20.301276 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.301235 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59685fb89b-vlj8s" podUID="b7888501-0547-435e-9d8f-115c803e7aa1" containerName="console" containerID="cri-o://67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74" gracePeriod=15 Jul 22 16:29:20.390299 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.390274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hm6sf" Jul 22 16:29:20.556328 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.556267 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:29:20.577471 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-oauth-serving-cert\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.577754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577487 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-288ll\" (UniqueName: \"kubernetes.io/projected/b7888501-0547-435e-9d8f-115c803e7aa1-kube-api-access-288ll\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.577754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-console-config\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.577754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-serving-cert\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.577754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577752 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-trusted-ca-bundle\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.578084 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577845 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-oauth-config\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.578084 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-service-ca\") pod \"b7888501-0547-435e-9d8f-115c803e7aa1\" (UID: \"b7888501-0547-435e-9d8f-115c803e7aa1\") " Jul 22 16:29:20.578084 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.577962 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:29:20.578813 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.578427 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-oauth-serving-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:20.578813 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.578676 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-console-config" (OuterVolumeSpecName: "console-config") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:29:20.579022 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.578866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:29:20.579022 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.578926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:29:20.580193 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.580169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7888501-0547-435e-9d8f-115c803e7aa1-kube-api-access-288ll" (OuterVolumeSpecName: "kube-api-access-288ll") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "kube-api-access-288ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:20.580611 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.580590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:29:20.581028 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.581008 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7888501-0547-435e-9d8f-115c803e7aa1" (UID: "b7888501-0547-435e-9d8f-115c803e7aa1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:29:20.679717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.679677 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-oauth-config\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:20.679717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.679710 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:20.679717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.679719 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288ll\" (UniqueName: \"kubernetes.io/projected/b7888501-0547-435e-9d8f-115c803e7aa1-kube-api-access-288ll\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:20.679717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.679728 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-console-config\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:20.679979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.679738 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7888501-0547-435e-9d8f-115c803e7aa1-console-serving-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:20.679979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:20.679746 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7888501-0547-435e-9d8f-115c803e7aa1-trusted-ca-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:21.374144 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.374049 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7888501-0547-435e-9d8f-115c803e7aa1" containerID="67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74" exitCode=0 Jul 22 16:29:21.374144 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.374097 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59685fb89b-vlj8s" Jul 22 16:29:21.374144 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.374130 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59685fb89b-vlj8s" event={"ID":"b7888501-0547-435e-9d8f-115c803e7aa1","Type":"ContainerDied","Data":"67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74"} Jul 22 16:29:21.374717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.374175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59685fb89b-vlj8s" event={"ID":"b7888501-0547-435e-9d8f-115c803e7aa1","Type":"ContainerDied","Data":"47c79d6fbb97a4b78a4792f5924c4262633856c160da319e5322e1612aa71ee8"} Jul 22 16:29:21.374717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.374195 2580 scope.go:117] "RemoveContainer" containerID="67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74" Jul 22 16:29:21.382651 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.382624 2580 scope.go:117] "RemoveContainer" containerID="67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74" Jul 22 16:29:21.382909 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:21.382881 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74\": container with ID starting with 67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74 not found: ID does not exist" containerID="67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74" Jul 22 16:29:21.382969 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.382908 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74"} err="failed to get container status \"67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74\": rpc error: code = NotFound desc = could not find container \"67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74\": container with ID starting with 67c541563f0b76262aa5e89603605c6483b57194637e60db14f0a78e2c3e6c74 not found: ID does not exist" Jul 22 16:29:21.395951 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.395928 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59685fb89b-vlj8s"] Jul 22 16:29:21.400546 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:21.400526 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59685fb89b-vlj8s"] Jul 22 16:29:22.979609 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:22.979573 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7888501-0547-435e-9d8f-115c803e7aa1" path="/var/lib/kubelet/pods/b7888501-0547-435e-9d8f-115c803e7aa1/volumes" Jul 22 16:29:24.363717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.363684 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz"] Jul 22 16:29:24.364155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.364098 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7888501-0547-435e-9d8f-115c803e7aa1" containerName="console" Jul 22 16:29:24.364155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.364112 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7888501-0547-435e-9d8f-115c803e7aa1" containerName="console" Jul 22 16:29:24.364155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.364121 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0855483d-f380-4a59-8c37-4925781eeda3" containerName="registry-server" Jul 22 16:29:24.364155 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.364127 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0855483d-f380-4a59-8c37-4925781eeda3" containerName="registry-server" Jul 22 16:29:24.364285 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.364181 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0855483d-f380-4a59-8c37-4925781eeda3" containerName="registry-server" Jul 22 16:29:24.364285 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.364190 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7888501-0547-435e-9d8f-115c803e7aa1" containerName="console" Jul 22 16:29:24.368388 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.368371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.370994 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.370972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l6x57\"" Jul 22 16:29:24.376146 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.376122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz"] Jul 22 16:29:24.410999 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.410959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.410999 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.411005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9d2d\" (UniqueName: \"kubernetes.io/projected/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-kube-api-access-b9d2d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.411219 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.411038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.512200 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.512165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.512200 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.512204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9d2d\" (UniqueName: \"kubernetes.io/projected/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-kube-api-access-b9d2d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.512470 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.512240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.512592 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.512575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.512860 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.512650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.521175 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.521153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9d2d\" (UniqueName: \"kubernetes.io/projected/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-kube-api-access-b9d2d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.678717 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.678676 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:24.763748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.763347 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v"] Jul 22 16:29:24.769134 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.769106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.775703 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.775634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v"] Jul 22 16:29:24.813804 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.813768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.813972 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.813877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9n8\" (UniqueName: \"kubernetes.io/projected/e1902729-34b5-4f83-8bea-f74925cef10d-kube-api-access-pf9n8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.813972 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.813924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.817985 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.817959 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz"] Jul 22 16:29:24.819511 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:24.819481 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3904e122_3f53_4d22_b31c_f7e9dbb76cc6.slice/crio-9acecab0e4423fc2c026f7578c54bf4b2daafdb050ef8b5d0dae00afef50741d WatchSource:0}: Error finding container 9acecab0e4423fc2c026f7578c54bf4b2daafdb050ef8b5d0dae00afef50741d: Status 404 returned error can't find the container with id 9acecab0e4423fc2c026f7578c54bf4b2daafdb050ef8b5d0dae00afef50741d Jul 22 16:29:24.915277 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.915178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9n8\" (UniqueName: \"kubernetes.io/projected/e1902729-34b5-4f83-8bea-f74925cef10d-kube-api-access-pf9n8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.915277 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.915264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.915530 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.915332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.915723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.915700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.915791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.915730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:24.924127 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:24.924099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9n8\" (UniqueName: \"kubernetes.io/projected/e1902729-34b5-4f83-8bea-f74925cef10d-kube-api-access-pf9n8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:25.083459 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.083418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:25.167067 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.167037 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw"] Jul 22 16:29:25.172006 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.171986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.182424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.182381 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw"] Jul 22 16:29:25.217645 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.217612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.217800 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.217667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.217800 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.217743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccszj\" (UniqueName: \"kubernetes.io/projected/ab600634-fcc4-483e-8352-67c95a4fda9a-kube-api-access-ccszj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.223818 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.223795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v"] Jul 22 16:29:25.225686 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:25.225663 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1902729_34b5_4f83_8bea_f74925cef10d.slice/crio-26a7734c248bf07be5fae9a44c59d782ea97ab98852e7789d944e540fc422ac8 WatchSource:0}: Error finding container 26a7734c248bf07be5fae9a44c59d782ea97ab98852e7789d944e540fc422ac8: Status 404 returned error can't find the container with id 26a7734c248bf07be5fae9a44c59d782ea97ab98852e7789d944e540fc422ac8 Jul 22 16:29:25.318628 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.318598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.318764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.318661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.318764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.318696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccszj\" (UniqueName: \"kubernetes.io/projected/ab600634-fcc4-483e-8352-67c95a4fda9a-kube-api-access-ccszj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.318973 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.318954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.319014 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.319002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.328405 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.328373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccszj\" (UniqueName: \"kubernetes.io/projected/ab600634-fcc4-483e-8352-67c95a4fda9a-kube-api-access-ccszj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.398610 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.398574 2580 generic.go:356] "Generic (PLEG): container finished" podID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerID="cc5e799420585fa9ab4ee910f232a90aec41bee313c94580aa6740e63f469e13" exitCode=0 Jul 22 16:29:25.399059 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.398668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" event={"ID":"3904e122-3f53-4d22-b31c-f7e9dbb76cc6","Type":"ContainerDied","Data":"cc5e799420585fa9ab4ee910f232a90aec41bee313c94580aa6740e63f469e13"} Jul 22 16:29:25.399059 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.398739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" event={"ID":"3904e122-3f53-4d22-b31c-f7e9dbb76cc6","Type":"ContainerStarted","Data":"9acecab0e4423fc2c026f7578c54bf4b2daafdb050ef8b5d0dae00afef50741d"} Jul 22 16:29:25.400297 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.400275 2580 generic.go:356] "Generic (PLEG): container finished" podID="e1902729-34b5-4f83-8bea-f74925cef10d" containerID="905800e32f55efc2fa43e7d80bd3f9078fb72dc176cddfcd1d96c616fe5d3315" exitCode=0 Jul 22 16:29:25.400421 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.400339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" event={"ID":"e1902729-34b5-4f83-8bea-f74925cef10d","Type":"ContainerDied","Data":"905800e32f55efc2fa43e7d80bd3f9078fb72dc176cddfcd1d96c616fe5d3315"} Jul 22 16:29:25.400421 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.400378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" event={"ID":"e1902729-34b5-4f83-8bea-f74925cef10d","Type":"ContainerStarted","Data":"26a7734c248bf07be5fae9a44c59d782ea97ab98852e7789d944e540fc422ac8"} Jul 22 16:29:25.483697 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.483607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:25.566979 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.566944 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq"] Jul 22 16:29:25.572631 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.572608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.580583 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.580558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq"] Jul 22 16:29:25.620593 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.620562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.620593 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.620597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.620791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.620614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2snm\" (UniqueName: \"kubernetes.io/projected/790d3a30-f612-416c-8bfc-fa90bcaa42a3-kube-api-access-v2snm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.639181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.639150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw"] Jul 22 16:29:25.640546 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:25.640521 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab600634_fcc4_483e_8352_67c95a4fda9a.slice/crio-d6f0e1b248d3cef26c686ad703de1cee5d23b005190d2b729830445ab727dbfb WatchSource:0}: Error finding container d6f0e1b248d3cef26c686ad703de1cee5d23b005190d2b729830445ab727dbfb: Status 404 returned error can't find the container with id d6f0e1b248d3cef26c686ad703de1cee5d23b005190d2b729830445ab727dbfb Jul 22 16:29:25.722205 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.722173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.722366 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.722211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.722366 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.722235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2snm\" (UniqueName: \"kubernetes.io/projected/790d3a30-f612-416c-8bfc-fa90bcaa42a3-kube-api-access-v2snm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.722748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.722710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.722748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.722729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.733007 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.732973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2snm\" (UniqueName: \"kubernetes.io/projected/790d3a30-f612-416c-8bfc-fa90bcaa42a3-kube-api-access-v2snm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:25.883387 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:25.883354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:26.019109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.019050 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq"] Jul 22 16:29:26.026192 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:26.026160 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790d3a30_f612_416c_8bfc_fa90bcaa42a3.slice/crio-f7b51ffa83fe75a1b8013fe83ab070b3f5a4800e681212f74fcabced3d9b7a55 WatchSource:0}: Error finding container f7b51ffa83fe75a1b8013fe83ab070b3f5a4800e681212f74fcabced3d9b7a55: Status 404 returned error can't find the container with id f7b51ffa83fe75a1b8013fe83ab070b3f5a4800e681212f74fcabced3d9b7a55 Jul 22 16:29:26.406955 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.406924 2580 generic.go:356] "Generic (PLEG): container finished" podID="e1902729-34b5-4f83-8bea-f74925cef10d" containerID="746987e34402068db2e580aaba6a17ee14d56dcf036b34a41cb8f8911d40fb31" exitCode=0 Jul 22 16:29:26.407373 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.407002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" event={"ID":"e1902729-34b5-4f83-8bea-f74925cef10d","Type":"ContainerDied","Data":"746987e34402068db2e580aaba6a17ee14d56dcf036b34a41cb8f8911d40fb31"} Jul 22 16:29:26.408610 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.408588 2580 generic.go:356] "Generic (PLEG): container finished" podID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerID="d78fac2648ea0c51f83fbd0152bc4962fe5b8b5a2eadbfc09734763b3d12bada" exitCode=0 Jul 22 16:29:26.408694 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.408667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" event={"ID":"790d3a30-f612-416c-8bfc-fa90bcaa42a3","Type":"ContainerDied","Data":"d78fac2648ea0c51f83fbd0152bc4962fe5b8b5a2eadbfc09734763b3d12bada"} Jul 22 16:29:26.408756 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.408713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" event={"ID":"790d3a30-f612-416c-8bfc-fa90bcaa42a3","Type":"ContainerStarted","Data":"f7b51ffa83fe75a1b8013fe83ab070b3f5a4800e681212f74fcabced3d9b7a55"} Jul 22 16:29:26.410705 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.410681 2580 generic.go:356] "Generic (PLEG): container finished" podID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerID="50b3b53d1ba6bc60bf7fc1b30eae3a7ffe923105a252876bd0a884645b61fd51" exitCode=0 Jul 22 16:29:26.410814 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.410768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" event={"ID":"3904e122-3f53-4d22-b31c-f7e9dbb76cc6","Type":"ContainerDied","Data":"50b3b53d1ba6bc60bf7fc1b30eae3a7ffe923105a252876bd0a884645b61fd51"} Jul 22 16:29:26.412506 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.412429 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerID="1544e0accc35b91448cbf80af7588281a1b613cfdfe661f89d03087da6b2db89" exitCode=0 Jul 22 16:29:26.412506 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.412484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" event={"ID":"ab600634-fcc4-483e-8352-67c95a4fda9a","Type":"ContainerDied","Data":"1544e0accc35b91448cbf80af7588281a1b613cfdfe661f89d03087da6b2db89"} Jul 22 16:29:26.412638 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:26.412523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" event={"ID":"ab600634-fcc4-483e-8352-67c95a4fda9a","Type":"ContainerStarted","Data":"d6f0e1b248d3cef26c686ad703de1cee5d23b005190d2b729830445ab727dbfb"} Jul 22 16:29:27.419466 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.419434 2580 generic.go:356] "Generic (PLEG): container finished" podID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerID="29502ad99702dcc1cce2ba706a80d2488597264a090bdc8f82ded828280cde73" exitCode=0 Jul 22 16:29:27.419842 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.419514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" event={"ID":"3904e122-3f53-4d22-b31c-f7e9dbb76cc6","Type":"ContainerDied","Data":"29502ad99702dcc1cce2ba706a80d2488597264a090bdc8f82ded828280cde73"} Jul 22 16:29:27.421317 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.421297 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerID="634fa772aeccbc88a807c086944aa04d14963aee2d3b262c815de69fb8949197" exitCode=0 Jul 22 16:29:27.421422 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.421380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" event={"ID":"ab600634-fcc4-483e-8352-67c95a4fda9a","Type":"ContainerDied","Data":"634fa772aeccbc88a807c086944aa04d14963aee2d3b262c815de69fb8949197"} Jul 22 16:29:27.423528 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.423507 2580 generic.go:356] "Generic (PLEG): container finished" podID="e1902729-34b5-4f83-8bea-f74925cef10d" containerID="f0c0740507b1b3b346436b983a688ce6f4a7a0ab8c37c06fc23fbebafc60883d" exitCode=0 Jul 22 16:29:27.423664 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.423596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" event={"ID":"e1902729-34b5-4f83-8bea-f74925cef10d","Type":"ContainerDied","Data":"f0c0740507b1b3b346436b983a688ce6f4a7a0ab8c37c06fc23fbebafc60883d"} Jul 22 16:29:27.425288 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.425267 2580 generic.go:356] "Generic (PLEG): container finished" podID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerID="d815a87042858365923672913bef6858454bd99eee91ce26572a40647752de8a" exitCode=0 Jul 22 16:29:27.425360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:27.425296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" event={"ID":"790d3a30-f612-416c-8bfc-fa90bcaa42a3","Type":"ContainerDied","Data":"d815a87042858365923672913bef6858454bd99eee91ce26572a40647752de8a"} Jul 22 16:29:28.432624 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.432594 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerID="3392e40be31a566cbe915294929f5e76f5d64beddf7260d8b34b1b4d2defde8b" exitCode=0 Jul 22 16:29:28.433063 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.432685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" event={"ID":"ab600634-fcc4-483e-8352-67c95a4fda9a","Type":"ContainerDied","Data":"3392e40be31a566cbe915294929f5e76f5d64beddf7260d8b34b1b4d2defde8b"} Jul 22 16:29:28.434663 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.434636 2580 generic.go:356] "Generic (PLEG): container finished" podID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerID="34368a82a6d8162552baf6760bd54a90c748ab3725c6c23efe5da603596daf6e" exitCode=0 Jul 22 16:29:28.434765 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.434660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" event={"ID":"790d3a30-f612-416c-8bfc-fa90bcaa42a3","Type":"ContainerDied","Data":"34368a82a6d8162552baf6760bd54a90c748ab3725c6c23efe5da603596daf6e"} Jul 22 16:29:28.606158 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.606132 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:28.609060 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.609039 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:28.649326 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649293 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-bundle\") pod \"e1902729-34b5-4f83-8bea-f74925cef10d\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " Jul 22 16:29:28.649326 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649332 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf9n8\" (UniqueName: \"kubernetes.io/projected/e1902729-34b5-4f83-8bea-f74925cef10d-kube-api-access-pf9n8\") pod \"e1902729-34b5-4f83-8bea-f74925cef10d\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " Jul 22 16:29:28.649595 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649417 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9d2d\" (UniqueName: \"kubernetes.io/projected/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-kube-api-access-b9d2d\") pod \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " Jul 22 16:29:28.649595 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-util\") pod \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " Jul 22 16:29:28.649595 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649476 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-bundle\") pod \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\" (UID: \"3904e122-3f53-4d22-b31c-f7e9dbb76cc6\") " Jul 22 16:29:28.649595 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649498 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-util\") pod \"e1902729-34b5-4f83-8bea-f74925cef10d\" (UID: \"e1902729-34b5-4f83-8bea-f74925cef10d\") " Jul 22 16:29:28.650020 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.649923 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-bundle" (OuterVolumeSpecName: "bundle") pod "e1902729-34b5-4f83-8bea-f74925cef10d" (UID: "e1902729-34b5-4f83-8bea-f74925cef10d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:28.650118 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.650068 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-bundle" (OuterVolumeSpecName: "bundle") pod "3904e122-3f53-4d22-b31c-f7e9dbb76cc6" (UID: "3904e122-3f53-4d22-b31c-f7e9dbb76cc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:28.651590 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.651544 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-kube-api-access-b9d2d" (OuterVolumeSpecName: "kube-api-access-b9d2d") pod "3904e122-3f53-4d22-b31c-f7e9dbb76cc6" (UID: "3904e122-3f53-4d22-b31c-f7e9dbb76cc6"). InnerVolumeSpecName "kube-api-access-b9d2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:28.651980 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.651955 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1902729-34b5-4f83-8bea-f74925cef10d-kube-api-access-pf9n8" (OuterVolumeSpecName: "kube-api-access-pf9n8") pod "e1902729-34b5-4f83-8bea-f74925cef10d" (UID: "e1902729-34b5-4f83-8bea-f74925cef10d"). InnerVolumeSpecName "kube-api-access-pf9n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:28.655553 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.655529 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-util" (OuterVolumeSpecName: "util") pod "e1902729-34b5-4f83-8bea-f74925cef10d" (UID: "e1902729-34b5-4f83-8bea-f74925cef10d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:28.655683 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.655661 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-util" (OuterVolumeSpecName: "util") pod "3904e122-3f53-4d22-b31c-f7e9dbb76cc6" (UID: "3904e122-3f53-4d22-b31c-f7e9dbb76cc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:28.750897 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.750861 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9d2d\" (UniqueName: \"kubernetes.io/projected/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-kube-api-access-b9d2d\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:28.750897 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.750895 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:28.750897 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.750905 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3904e122-3f53-4d22-b31c-f7e9dbb76cc6-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:28.751129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.750913 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:28.751129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.750921 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1902729-34b5-4f83-8bea-f74925cef10d-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:28.751129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:28.750929 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf9n8\" (UniqueName: \"kubernetes.io/projected/e1902729-34b5-4f83-8bea-f74925cef10d-kube-api-access-pf9n8\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:29.441862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.441819 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" Jul 22 16:29:29.441862 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.441827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz" event={"ID":"3904e122-3f53-4d22-b31c-f7e9dbb76cc6","Type":"ContainerDied","Data":"9acecab0e4423fc2c026f7578c54bf4b2daafdb050ef8b5d0dae00afef50741d"} Jul 22 16:29:29.442312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.441883 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9acecab0e4423fc2c026f7578c54bf4b2daafdb050ef8b5d0dae00afef50741d" Jul 22 16:29:29.443917 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.443815 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" Jul 22 16:29:29.443917 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.443867 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v" event={"ID":"e1902729-34b5-4f83-8bea-f74925cef10d","Type":"ContainerDied","Data":"26a7734c248bf07be5fae9a44c59d782ea97ab98852e7789d944e540fc422ac8"} Jul 22 16:29:29.443917 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.443909 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a7734c248bf07be5fae9a44c59d782ea97ab98852e7789d944e540fc422ac8" Jul 22 16:29:29.568867 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.568836 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:29.614671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.614639 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:29.658973 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.658938 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-bundle\") pod \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " Jul 22 16:29:29.658973 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.658974 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2snm\" (UniqueName: \"kubernetes.io/projected/790d3a30-f612-416c-8bfc-fa90bcaa42a3-kube-api-access-v2snm\") pod \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " Jul 22 16:29:29.659164 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.659067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-util\") pod \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\" (UID: \"790d3a30-f612-416c-8bfc-fa90bcaa42a3\") " Jul 22 16:29:29.659164 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.659085 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-bundle\") pod \"ab600634-fcc4-483e-8352-67c95a4fda9a\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " Jul 22 16:29:29.659251 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.659201 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccszj\" (UniqueName: \"kubernetes.io/projected/ab600634-fcc4-483e-8352-67c95a4fda9a-kube-api-access-ccszj\") pod \"ab600634-fcc4-483e-8352-67c95a4fda9a\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " Jul 22 16:29:29.659251 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.659243 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-util\") pod \"ab600634-fcc4-483e-8352-67c95a4fda9a\" (UID: \"ab600634-fcc4-483e-8352-67c95a4fda9a\") " Jul 22 16:29:29.659730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.659700 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-bundle" (OuterVolumeSpecName: "bundle") pod "790d3a30-f612-416c-8bfc-fa90bcaa42a3" (UID: "790d3a30-f612-416c-8bfc-fa90bcaa42a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:29.660040 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.660014 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-bundle" (OuterVolumeSpecName: "bundle") pod "ab600634-fcc4-483e-8352-67c95a4fda9a" (UID: "ab600634-fcc4-483e-8352-67c95a4fda9a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:29.661491 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.661465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790d3a30-f612-416c-8bfc-fa90bcaa42a3-kube-api-access-v2snm" (OuterVolumeSpecName: "kube-api-access-v2snm") pod "790d3a30-f612-416c-8bfc-fa90bcaa42a3" (UID: "790d3a30-f612-416c-8bfc-fa90bcaa42a3"). InnerVolumeSpecName "kube-api-access-v2snm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:29.661593 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.661502 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab600634-fcc4-483e-8352-67c95a4fda9a-kube-api-access-ccszj" (OuterVolumeSpecName: "kube-api-access-ccszj") pod "ab600634-fcc4-483e-8352-67c95a4fda9a" (UID: "ab600634-fcc4-483e-8352-67c95a4fda9a"). InnerVolumeSpecName "kube-api-access-ccszj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:29.666230 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.666204 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-util" (OuterVolumeSpecName: "util") pod "790d3a30-f612-416c-8bfc-fa90bcaa42a3" (UID: "790d3a30-f612-416c-8bfc-fa90bcaa42a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:29.666968 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.666952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-util" (OuterVolumeSpecName: "util") pod "ab600634-fcc4-483e-8352-67c95a4fda9a" (UID: "ab600634-fcc4-483e-8352-67c95a4fda9a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:29.760625 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.760572 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:29.766438 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.760620 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:29.766438 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.761074 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccszj\" (UniqueName: \"kubernetes.io/projected/ab600634-fcc4-483e-8352-67c95a4fda9a-kube-api-access-ccszj\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:29.766438 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.761093 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab600634-fcc4-483e-8352-67c95a4fda9a-util\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:29.766438 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.761154 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790d3a30-f612-416c-8bfc-fa90bcaa42a3-bundle\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:29.766438 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:29.761200 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2snm\" (UniqueName: \"kubernetes.io/projected/790d3a30-f612-416c-8bfc-fa90bcaa42a3-kube-api-access-v2snm\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:30.452158 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:30.452131 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" Jul 22 16:29:30.452623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:30.452126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw" event={"ID":"ab600634-fcc4-483e-8352-67c95a4fda9a","Type":"ContainerDied","Data":"d6f0e1b248d3cef26c686ad703de1cee5d23b005190d2b729830445ab727dbfb"} Jul 22 16:29:30.452623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:30.452251 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f0e1b248d3cef26c686ad703de1cee5d23b005190d2b729830445ab727dbfb" Jul 22 16:29:30.454121 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:30.454100 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" Jul 22 16:29:30.454209 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:30.454116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq" event={"ID":"790d3a30-f612-416c-8bfc-fa90bcaa42a3","Type":"ContainerDied","Data":"f7b51ffa83fe75a1b8013fe83ab070b3f5a4800e681212f74fcabced3d9b7a55"} Jul 22 16:29:30.454209 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:30.454166 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b51ffa83fe75a1b8013fe83ab070b3f5a4800e681212f74fcabced3d9b7a55" Jul 22 16:29:36.329456 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329346 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt"] Jul 22 16:29:36.329810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329769 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="util" Jul 22 16:29:36.329810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329782 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="util" Jul 22 16:29:36.329810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329789 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="pull" Jul 22 16:29:36.329810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329795 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="pull" Jul 22 16:29:36.329810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329802 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="pull" Jul 22 16:29:36.329810 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329807 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="pull" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329815 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="util" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329820 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="util" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329834 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329839 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329844 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329858 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329865 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329870 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329878 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329882 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329891 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="pull" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329895 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="pull" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329905 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="pull" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329910 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="pull" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329920 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="util" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329925 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="util" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329931 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="util" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329935 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="util" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329987 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab600634-fcc4-483e-8352-67c95a4fda9a" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.329996 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1902729-34b5-4f83-8bea-f74925cef10d" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.330005 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3904e122-3f53-4d22-b31c-f7e9dbb76cc6" containerName="extract" Jul 22 16:29:36.330003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.330013 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="790d3a30-f612-416c-8bfc-fa90bcaa42a3" containerName="extract" Jul 22 16:29:36.334348 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.334329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.336769 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.336750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mqv9r\"" Jul 22 16:29:36.348152 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.348124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt"] Jul 22 16:29:36.419785 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.419747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbmz\" (UniqueName: \"kubernetes.io/projected/8fa351ca-074b-4679-ab84-e07561452b53-kube-api-access-5lbmz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.419952 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.419793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fa351ca-074b-4679-ab84-e07561452b53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.520412 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.520352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbmz\" (UniqueName: \"kubernetes.io/projected/8fa351ca-074b-4679-ab84-e07561452b53-kube-api-access-5lbmz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.520608 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.520429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fa351ca-074b-4679-ab84-e07561452b53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.521574 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.521551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fa351ca-074b-4679-ab84-e07561452b53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.529470 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.529445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbmz\" (UniqueName: \"kubernetes.io/projected/8fa351ca-074b-4679-ab84-e07561452b53-kube-api-access-5lbmz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.644167 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.644073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:36.788197 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:36.788171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt"] Jul 22 16:29:36.788612 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:36.788581 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa351ca_074b_4679_ab84_e07561452b53.slice/crio-f5f515029141c7979005b9227ecfd1c7c117f1d3389d333e9e208ffcdf50028f WatchSource:0}: Error finding container f5f515029141c7979005b9227ecfd1c7c117f1d3389d333e9e208ffcdf50028f: Status 404 returned error can't find the container with id f5f515029141c7979005b9227ecfd1c7c117f1d3389d333e9e208ffcdf50028f Jul 22 16:29:37.495805 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:37.495768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" event={"ID":"8fa351ca-074b-4679-ab84-e07561452b53","Type":"ContainerStarted","Data":"f5f515029141c7979005b9227ecfd1c7c117f1d3389d333e9e208ffcdf50028f"} Jul 22 16:29:38.987495 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:38.987465 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr"] Jul 22 16:29:38.990938 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:38.990920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:38.993355 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:38.993335 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m8qlj\"" Jul 22 16:29:39.002065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.002038 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr"] Jul 22 16:29:39.045927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.045883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rslzp\" (UniqueName: \"kubernetes.io/projected/10be3612-5a60-4a91-adc9-ae82b4181ce4-kube-api-access-rslzp\") pod \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" (UID: \"10be3612-5a60-4a91-adc9-ae82b4181ce4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:39.146668 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.146631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rslzp\" (UniqueName: \"kubernetes.io/projected/10be3612-5a60-4a91-adc9-ae82b4181ce4-kube-api-access-rslzp\") pod \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" (UID: \"10be3612-5a60-4a91-adc9-ae82b4181ce4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:39.156831 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.156802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rslzp\" (UniqueName: \"kubernetes.io/projected/10be3612-5a60-4a91-adc9-ae82b4181ce4-kube-api-access-rslzp\") pod \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" (UID: \"10be3612-5a60-4a91-adc9-ae82b4181ce4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:39.302087 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.301986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:39.457673 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.457644 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr"] Jul 22 16:29:39.508025 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:39.507995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" event={"ID":"10be3612-5a60-4a91-adc9-ae82b4181ce4","Type":"ContainerStarted","Data":"e6bb27363fd30e0b8b0302a0f6bf5a2033807dbafe111f3be4ba13d1d19bf601"} Jul 22 16:29:41.444628 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.444545 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ppfns"] Jul 22 16:29:41.448526 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.448505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:41.456388 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.456365 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xjfq4\"" Jul 22 16:29:41.481448 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.481276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ppfns"] Jul 22 16:29:41.524719 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.524673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" event={"ID":"10be3612-5a60-4a91-adc9-ae82b4181ce4","Type":"ContainerStarted","Data":"884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f"} Jul 22 16:29:41.524917 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.524820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:41.565758 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.565697 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" podStartSLOduration=2.010673931 podStartE2EDuration="3.565675818s" totalImagesPullingTime="1.555001887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:41.564666436 +0000 UTC m=+539.144760360" watchObservedRunningTime="2026-07-22 16:29:41.565675818 +0000 UTC m=+539.145769714" Jul 22 16:29:41.572101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.572065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q49m5\" (UniqueName: \"kubernetes.io/projected/c77d0333-350a-4653-8287-f3aadf958258-kube-api-access-q49m5\") pod \"authorino-operator-75cc8d5cb7-ppfns\" (UID: \"c77d0333-350a-4653-8287-f3aadf958258\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:41.673163 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.673113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q49m5\" (UniqueName: \"kubernetes.io/projected/c77d0333-350a-4653-8287-f3aadf958258-kube-api-access-q49m5\") pod \"authorino-operator-75cc8d5cb7-ppfns\" (UID: \"c77d0333-350a-4653-8287-f3aadf958258\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:41.684503 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.684471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q49m5\" (UniqueName: \"kubernetes.io/projected/c77d0333-350a-4653-8287-f3aadf958258-kube-api-access-q49m5\") pod \"authorino-operator-75cc8d5cb7-ppfns\" (UID: \"c77d0333-350a-4653-8287-f3aadf958258\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:41.761123 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:41.761083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:42.528990 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:42.528954 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ppfns"] Jul 22 16:29:42.530945 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:42.530914 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77d0333_350a_4653_8287_f3aadf958258.slice/crio-cfb18089fd29051f70bbee766b1902e2053d129b1e71543de187e6468770ef59 WatchSource:0}: Error finding container cfb18089fd29051f70bbee766b1902e2053d129b1e71543de187e6468770ef59: Status 404 returned error can't find the container with id cfb18089fd29051f70bbee766b1902e2053d129b1e71543de187e6468770ef59 Jul 22 16:29:43.545877 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:43.545655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" event={"ID":"8fa351ca-074b-4679-ab84-e07561452b53","Type":"ContainerStarted","Data":"51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959"} Jul 22 16:29:43.545877 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:43.545763 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:43.547775 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:43.547705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" event={"ID":"c77d0333-350a-4653-8287-f3aadf958258","Type":"ContainerStarted","Data":"cfb18089fd29051f70bbee766b1902e2053d129b1e71543de187e6468770ef59"} Jul 22 16:29:43.587470 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:43.587406 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" podStartSLOduration=1.927851252 podStartE2EDuration="7.587374994s" totalImagesPullingTime="5.659523742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:43.585739817 +0000 UTC m=+541.165833714" watchObservedRunningTime="2026-07-22 16:29:43.587374994 +0000 UTC m=+541.167468890" Jul 22 16:29:45.561450 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:45.561418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" event={"ID":"c77d0333-350a-4653-8287-f3aadf958258","Type":"ContainerStarted","Data":"c58e81c13bf58281db2c4b4a8a3730263c7f40106a27e0269e0d01ebe409a390"} Jul 22 16:29:45.561832 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:45.561491 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:45.580962 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:45.580907 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" podStartSLOduration=2.270248402 podStartE2EDuration="4.580890009s" totalImagesPullingTime="2.310641607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:45.578681041 +0000 UTC m=+543.158774963" watchObservedRunningTime="2026-07-22 16:29:45.580890009 +0000 UTC m=+543.160983905" Jul 22 16:29:52.536884 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:52.536848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:54.556815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:54.556783 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.523920 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.523878 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt"] Jul 22 16:29:56.524314 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.524095 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" podUID="8fa351ca-074b-4679-ab84-e07561452b53" containerName="manager" containerID="cri-o://51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959" gracePeriod=2 Jul 22 16:29:56.533737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.533704 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt"] Jul 22 16:29:56.535363 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.535332 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8fa351ca-074b-4679-ab84-e07561452b53\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\": pods \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.546799 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.546771 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc"] Jul 22 16:29:56.547313 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.547285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa351ca-074b-4679-ab84-e07561452b53" containerName="manager" Jul 22 16:29:56.547313 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.547308 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa351ca-074b-4679-ab84-e07561452b53" containerName="manager" Jul 22 16:29:56.547489 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.547453 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa351ca-074b-4679-ab84-e07561452b53" containerName="manager" Jul 22 16:29:56.550522 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.550505 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr"] Jul 22 16:29:56.550680 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.550658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.550680 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.550664 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" containerName="manager" containerID="cri-o://884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f" gracePeriod=2 Jul 22 16:29:56.552908 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.552876 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="8fa351ca-074b-4679-ab84-e07561452b53" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.559129 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.559107 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr"] Jul 22 16:29:56.560814 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.560789 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="8fa351ca-074b-4679-ab84-e07561452b53" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.563204 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.563171 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:56.565010 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.564988 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc"] Jul 22 16:29:56.569642 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.569624 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ppfns" Jul 22 16:29:56.577170 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.577148 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m"] Jul 22 16:29:56.577570 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.577557 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" containerName="manager" Jul 22 16:29:56.577616 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.577572 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" containerName="manager" Jul 22 16:29:56.577658 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.577649 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" containerName="manager" Jul 22 16:29:56.580719 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.580702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:29:56.587000 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.586971 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="8fa351ca-074b-4679-ab84-e07561452b53" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.594607 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.594506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m"] Jul 22 16:29:56.641570 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.641529 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:56.713799 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.713758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srt4m\" (UniqueName: \"kubernetes.io/projected/961ac820-3682-452a-a58b-d920d2bf1376-kube-api-access-srt4m\") pod \"limitador-operator-controller-manager-859d7b78bd-q744m\" (UID: \"961ac820-3682-452a-a58b-d920d2bf1376\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:29:56.714066 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.714044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gqp\" (UniqueName: \"kubernetes.io/projected/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-kube-api-access-j6gqp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7x9vc\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.714208 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.714192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7x9vc\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.799619 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.799590 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.801962 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.801939 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:56.802434 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.802416 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:56.805447 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.805425 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="8fa351ca-074b-4679-ab84-e07561452b53" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.807219 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.807198 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rgkfr\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:56.809022 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:56.809001 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s8nlt\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="8fa351ca-074b-4679-ab84-e07561452b53" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:56.815479 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.815457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srt4m\" (UniqueName: \"kubernetes.io/projected/961ac820-3682-452a-a58b-d920d2bf1376-kube-api-access-srt4m\") pod \"limitador-operator-controller-manager-859d7b78bd-q744m\" (UID: \"961ac820-3682-452a-a58b-d920d2bf1376\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:29:56.815569 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.815492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gqp\" (UniqueName: \"kubernetes.io/projected/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-kube-api-access-j6gqp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7x9vc\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.815569 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.815511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7x9vc\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.815866 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.815853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7x9vc\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.825432 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.825384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gqp\" (UniqueName: \"kubernetes.io/projected/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-kube-api-access-j6gqp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7x9vc\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.825634 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.825616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srt4m\" (UniqueName: \"kubernetes.io/projected/961ac820-3682-452a-a58b-d920d2bf1376-kube-api-access-srt4m\") pod \"limitador-operator-controller-manager-859d7b78bd-q744m\" (UID: \"961ac820-3682-452a-a58b-d920d2bf1376\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:29:56.916219 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.916169 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fa351ca-074b-4679-ab84-e07561452b53-extensions-socket-volume\") pod \"8fa351ca-074b-4679-ab84-e07561452b53\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " Jul 22 16:29:56.916444 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.916298 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lbmz\" (UniqueName: \"kubernetes.io/projected/8fa351ca-074b-4679-ab84-e07561452b53-kube-api-access-5lbmz\") pod \"8fa351ca-074b-4679-ab84-e07561452b53\" (UID: \"8fa351ca-074b-4679-ab84-e07561452b53\") " Jul 22 16:29:56.916444 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.916330 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rslzp\" (UniqueName: \"kubernetes.io/projected/10be3612-5a60-4a91-adc9-ae82b4181ce4-kube-api-access-rslzp\") pod \"10be3612-5a60-4a91-adc9-ae82b4181ce4\" (UID: \"10be3612-5a60-4a91-adc9-ae82b4181ce4\") " Jul 22 16:29:56.916678 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.916654 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa351ca-074b-4679-ab84-e07561452b53-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8fa351ca-074b-4679-ab84-e07561452b53" (UID: "8fa351ca-074b-4679-ab84-e07561452b53"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:29:56.916766 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.916753 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fa351ca-074b-4679-ab84-e07561452b53-extensions-socket-volume\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:56.918345 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.918323 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa351ca-074b-4679-ab84-e07561452b53-kube-api-access-5lbmz" (OuterVolumeSpecName: "kube-api-access-5lbmz") pod "8fa351ca-074b-4679-ab84-e07561452b53" (UID: "8fa351ca-074b-4679-ab84-e07561452b53"). InnerVolumeSpecName "kube-api-access-5lbmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:56.918431 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.918344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10be3612-5a60-4a91-adc9-ae82b4181ce4-kube-api-access-rslzp" (OuterVolumeSpecName: "kube-api-access-rslzp") pod "10be3612-5a60-4a91-adc9-ae82b4181ce4" (UID: "10be3612-5a60-4a91-adc9-ae82b4181ce4"). InnerVolumeSpecName "kube-api-access-rslzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:29:56.960421 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.960366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:56.966642 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.966613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:29:56.980670 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.980637 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10be3612-5a60-4a91-adc9-ae82b4181ce4" path="/var/lib/kubelet/pods/10be3612-5a60-4a91-adc9-ae82b4181ce4/volumes" Jul 22 16:29:56.981085 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:56.981066 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa351ca-074b-4679-ab84-e07561452b53" path="/var/lib/kubelet/pods/8fa351ca-074b-4679-ab84-e07561452b53/volumes" Jul 22 16:29:57.018221 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.018188 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lbmz\" (UniqueName: \"kubernetes.io/projected/8fa351ca-074b-4679-ab84-e07561452b53-kube-api-access-5lbmz\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:57.018221 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.018221 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rslzp\" (UniqueName: \"kubernetes.io/projected/10be3612-5a60-4a91-adc9-ae82b4181ce4-kube-api-access-rslzp\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:29:57.119330 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.119289 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc"] Jul 22 16:29:57.120940 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:57.120910 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3b3258_c978_4c87_851b_b0a5e8ac1ecf.slice/crio-4c6545afd029c960582a108acb514c0ecd2aad3ea366d0cdcfc7a7d39f8fefb3 WatchSource:0}: Error finding container 4c6545afd029c960582a108acb514c0ecd2aad3ea366d0cdcfc7a7d39f8fefb3: Status 404 returned error can't find the container with id 4c6545afd029c960582a108acb514c0ecd2aad3ea366d0cdcfc7a7d39f8fefb3 Jul 22 16:29:57.146580 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.146554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m"] Jul 22 16:29:57.155507 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:57.155477 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961ac820_3682_452a_a58b_d920d2bf1376.slice/crio-1b1a677d819bfa2ac5220420f1494e86d75e00a8ad6fbd8b2a81326ba8ed30bd WatchSource:0}: Error finding container 1b1a677d819bfa2ac5220420f1494e86d75e00a8ad6fbd8b2a81326ba8ed30bd: Status 404 returned error can't find the container with id 1b1a677d819bfa2ac5220420f1494e86d75e00a8ad6fbd8b2a81326ba8ed30bd Jul 22 16:29:57.638791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.638691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" event={"ID":"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf","Type":"ContainerStarted","Data":"a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779"} Jul 22 16:29:57.638791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.638788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" event={"ID":"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf","Type":"ContainerStarted","Data":"4c6545afd029c960582a108acb514c0ecd2aad3ea366d0cdcfc7a7d39f8fefb3"} Jul 22 16:29:57.639297 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.638817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:29:57.640161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.640132 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fa351ca-074b-4679-ab84-e07561452b53" containerID="51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959" exitCode=0 Jul 22 16:29:57.640271 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.640176 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s8nlt" Jul 22 16:29:57.640271 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.640215 2580 scope.go:117] "RemoveContainer" containerID="51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959" Jul 22 16:29:57.641746 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.641723 2580 generic.go:356] "Generic (PLEG): container finished" podID="10be3612-5a60-4a91-adc9-ae82b4181ce4" containerID="884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f" exitCode=0 Jul 22 16:29:57.641847 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.641793 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rgkfr" Jul 22 16:29:57.644180 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.644154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" event={"ID":"961ac820-3682-452a-a58b-d920d2bf1376","Type":"ContainerStarted","Data":"7c73b960e403ba617b6431caf5cb174c69588299d66e007d265e9063732dc1ea"} Jul 22 16:29:57.644289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.644253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" event={"ID":"961ac820-3682-452a-a58b-d920d2bf1376","Type":"ContainerStarted","Data":"1b1a677d819bfa2ac5220420f1494e86d75e00a8ad6fbd8b2a81326ba8ed30bd"} Jul 22 16:29:57.644372 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.644353 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:29:57.650212 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.650194 2580 scope.go:117] "RemoveContainer" containerID="51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959" Jul 22 16:29:57.650484 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:57.650461 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959\": container with ID starting with 51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959 not found: ID does not exist" containerID="51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959" Jul 22 16:29:57.650547 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.650493 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959"} err="failed to get container status \"51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959\": rpc error: code = NotFound desc = could not find container \"51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959\": container with ID starting with 51273b09a4ebb27cf4b2e2211c3001dbb63813419c8fcaf9a5df69d2c1b63959 not found: ID does not exist" Jul 22 16:29:57.650547 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.650509 2580 scope.go:117] "RemoveContainer" containerID="884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f" Jul 22 16:29:57.658764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.658749 2580 scope.go:117] "RemoveContainer" containerID="884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f" Jul 22 16:29:57.659019 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:29:57.659002 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f\": container with ID starting with 884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f not found: ID does not exist" containerID="884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f" Jul 22 16:29:57.659074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.659024 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f"} err="failed to get container status \"884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f\": rpc error: code = NotFound desc = could not find container \"884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f\": container with ID starting with 884e9c8e940c3b8a55e64b1f1bf2eef88de596f324b37b2fe22e88840c60d18f not found: ID does not exist" Jul 22 16:29:57.662037 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.661997 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" podStartSLOduration=1.6619806929999998 podStartE2EDuration="1.661980693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:57.66158469 +0000 UTC m=+555.241678588" watchObservedRunningTime="2026-07-22 16:29:57.661980693 +0000 UTC m=+555.242074573" Jul 22 16:29:57.684655 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.684610 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" podStartSLOduration=1.684595168 podStartE2EDuration="1.684595168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:57.682042098 +0000 UTC m=+555.262135994" watchObservedRunningTime="2026-07-22 16:29:57.684595168 +0000 UTC m=+555.264689065" Jul 22 16:29:57.905032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.904948 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm"] Jul 22 16:29:57.910044 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.910016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:57.925617 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:57.925591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm"] Jul 22 16:29:58.027729 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.027688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c8d142-3b71-4623-be8a-d258bbfda6ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-67kwm\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.027729 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.027735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7hb\" (UniqueName: \"kubernetes.io/projected/88c8d142-3b71-4623-be8a-d258bbfda6ca-kube-api-access-nt7hb\") pod \"kuadrant-operator-controller-manager-86878c769f-67kwm\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.128733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.128694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7hb\" (UniqueName: \"kubernetes.io/projected/88c8d142-3b71-4623-be8a-d258bbfda6ca-kube-api-access-nt7hb\") pod \"kuadrant-operator-controller-manager-86878c769f-67kwm\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.128940 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.128808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c8d142-3b71-4623-be8a-d258bbfda6ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-67kwm\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.129137 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.129119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c8d142-3b71-4623-be8a-d258bbfda6ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-67kwm\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.144770 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.144737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7hb\" (UniqueName: \"kubernetes.io/projected/88c8d142-3b71-4623-be8a-d258bbfda6ca-kube-api-access-nt7hb\") pod \"kuadrant-operator-controller-manager-86878c769f-67kwm\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.219865 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.219773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.370872 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.370846 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm"] Jul 22 16:29:58.372693 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:29:58.372673 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c8d142_3b71_4623_be8a_d258bbfda6ca.slice/crio-62ed89c0b6780093f8b8d039fdfaa5f46adf6c1c1967cf21965094c18ad86ec2 WatchSource:0}: Error finding container 62ed89c0b6780093f8b8d039fdfaa5f46adf6c1c1967cf21965094c18ad86ec2: Status 404 returned error can't find the container with id 62ed89c0b6780093f8b8d039fdfaa5f46adf6c1c1967cf21965094c18ad86ec2 Jul 22 16:29:58.652927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.652893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" event={"ID":"88c8d142-3b71-4623-be8a-d258bbfda6ca","Type":"ContainerStarted","Data":"5e94225f666ef86558e97a2b0970be24ca159053d0e65661a98bdc01e38ad31d"} Jul 22 16:29:58.653409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.652957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" event={"ID":"88c8d142-3b71-4623-be8a-d258bbfda6ca","Type":"ContainerStarted","Data":"62ed89c0b6780093f8b8d039fdfaa5f46adf6c1c1967cf21965094c18ad86ec2"} Jul 22 16:29:58.653409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.652984 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:29:58.684056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:29:58.684000 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" podStartSLOduration=1.683984739 podStartE2EDuration="1.683984739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:29:58.681426312 +0000 UTC m=+556.261520203" watchObservedRunningTime="2026-07-22 16:29:58.683984739 +0000 UTC m=+556.264078693" Jul 22 16:30:08.657182 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:08.657146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:30:08.657638 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:08.657265 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q744m" Jul 22 16:30:09.662951 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:09.662919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:30:09.721663 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:09.721624 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc"] Jul 22 16:30:09.721851 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:09.721823 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" podUID="ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" containerName="manager" containerID="cri-o://a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779" gracePeriod=10 Jul 22 16:30:09.994806 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:09.994784 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:30:10.025646 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.025605 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-extensions-socket-volume\") pod \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " Jul 22 16:30:10.025815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.025662 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6gqp\" (UniqueName: \"kubernetes.io/projected/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-kube-api-access-j6gqp\") pod \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\" (UID: \"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf\") " Jul 22 16:30:10.026008 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.025971 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" (UID: "ac3b3258-c978-4c87-851b-b0a5e8ac1ecf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:30:10.026146 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.026063 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-extensions-socket-volume\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:30:10.027951 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.027926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-kube-api-access-j6gqp" (OuterVolumeSpecName: "kube-api-access-j6gqp") pod "ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" (UID: "ac3b3258-c978-4c87-851b-b0a5e8ac1ecf"). InnerVolumeSpecName "kube-api-access-j6gqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:30:10.126494 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.126448 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6gqp\" (UniqueName: \"kubernetes.io/projected/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf-kube-api-access-j6gqp\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:30:10.726848 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.726806 2580 generic.go:356] "Generic (PLEG): container finished" podID="ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" containerID="a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779" exitCode=0 Jul 22 16:30:10.726848 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.726852 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" Jul 22 16:30:10.727344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.726846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" event={"ID":"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf","Type":"ContainerDied","Data":"a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779"} Jul 22 16:30:10.727344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.726907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc" event={"ID":"ac3b3258-c978-4c87-851b-b0a5e8ac1ecf","Type":"ContainerDied","Data":"4c6545afd029c960582a108acb514c0ecd2aad3ea366d0cdcfc7a7d39f8fefb3"} Jul 22 16:30:10.727344 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.726930 2580 scope.go:117] "RemoveContainer" containerID="a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779" Jul 22 16:30:10.737153 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.737126 2580 scope.go:117] "RemoveContainer" containerID="a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779" Jul 22 16:30:10.737511 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:30:10.737491 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779\": container with ID starting with a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779 not found: ID does not exist" containerID="a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779" Jul 22 16:30:10.737590 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.737521 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779"} err="failed to get container status \"a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779\": rpc error: code = NotFound desc = could not find container \"a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779\": container with ID starting with a54b5d4112fa1044a40f85219f90a684f01769ed2bfa41e7b685b3e2e6ba4779 not found: ID does not exist" Jul 22 16:30:10.752461 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.752422 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc"] Jul 22 16:30:10.759075 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.759041 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7x9vc"] Jul 22 16:30:10.986830 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:10.986727 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" path="/var/lib/kubelet/pods/ac3b3258-c978-4c87-851b-b0a5e8ac1ecf/volumes" Jul 22 16:30:28.673924 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.673883 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:28.674638 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.674616 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" containerName="manager" Jul 22 16:30:28.674730 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.674640 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" containerName="manager" Jul 22 16:30:28.674787 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.674764 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3b3258-c978-4c87-851b-b0a5e8ac1ecf" containerName="manager" Jul 22 16:30:28.683234 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.683175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.685864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.685829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:28.685864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.685855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l6x57\"" Jul 22 16:30:28.686180 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.685855 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 16:30:28.769283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.769233 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:28.804813 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.804760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-config-file\") pod \"limitador-limitador-64c94fd64f-fk6m4\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.804813 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.804813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55brr\" (UniqueName: \"kubernetes.io/projected/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-kube-api-access-55brr\") pod \"limitador-limitador-64c94fd64f-fk6m4\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.905878 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.905832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-config-file\") pod \"limitador-limitador-64c94fd64f-fk6m4\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.905878 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.905883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55brr\" (UniqueName: \"kubernetes.io/projected/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-kube-api-access-55brr\") pod \"limitador-limitador-64c94fd64f-fk6m4\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.906649 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.906629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-config-file\") pod \"limitador-limitador-64c94fd64f-fk6m4\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.915168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.915131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55brr\" (UniqueName: \"kubernetes.io/projected/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-kube-api-access-55brr\") pod \"limitador-limitador-64c94fd64f-fk6m4\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:28.995960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:28.995927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:29.163953 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.160936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:29.168732 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:30:29.168697 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9b51e0_94ae_4ec9_82c6_b60cb807da7c.slice/crio-a78c65e7d251861952cc5f2398a0152f2ddb8c0d41c0ce9bf37ac6f95afa7ee3 WatchSource:0}: Error finding container a78c65e7d251861952cc5f2398a0152f2ddb8c0d41c0ce9bf37ac6f95afa7ee3: Status 404 returned error can't find the container with id a78c65e7d251861952cc5f2398a0152f2ddb8c0d41c0ce9bf37ac6f95afa7ee3 Jul 22 16:30:29.677596 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.677559 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qfnwc"] Jul 22 16:30:29.682675 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.682645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:29.686227 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.686199 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tfq9z\"" Jul 22 16:30:29.692594 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.692558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qfnwc"] Jul 22 16:30:29.815694 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.815648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbrj\" (UniqueName: \"kubernetes.io/projected/f9692eb5-5359-4ed0-9d66-6b93b03f1692-kube-api-access-fsbrj\") pod \"authorino-58cd955675-qfnwc\" (UID: \"f9692eb5-5359-4ed0-9d66-6b93b03f1692\") " pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:29.846358 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.846296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" event={"ID":"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c","Type":"ContainerStarted","Data":"a78c65e7d251861952cc5f2398a0152f2ddb8c0d41c0ce9bf37ac6f95afa7ee3"} Jul 22 16:30:29.916715 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.916659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbrj\" (UniqueName: \"kubernetes.io/projected/f9692eb5-5359-4ed0-9d66-6b93b03f1692-kube-api-access-fsbrj\") pod \"authorino-58cd955675-qfnwc\" (UID: \"f9692eb5-5359-4ed0-9d66-6b93b03f1692\") " pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:29.930187 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.930046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbrj\" (UniqueName: \"kubernetes.io/projected/f9692eb5-5359-4ed0-9d66-6b93b03f1692-kube-api-access-fsbrj\") pod \"authorino-58cd955675-qfnwc\" (UID: \"f9692eb5-5359-4ed0-9d66-6b93b03f1692\") " pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:29.994115 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:29.994071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:30.198350 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:30.198313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qfnwc"] Jul 22 16:30:30.201161 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:30:30.201120 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9692eb5_5359_4ed0_9d66_6b93b03f1692.slice/crio-87847637c2bfb8270d08a8cc3850dfb053d0f2f1655dd9d1c6c5ff9631587458 WatchSource:0}: Error finding container 87847637c2bfb8270d08a8cc3850dfb053d0f2f1655dd9d1c6c5ff9631587458: Status 404 returned error can't find the container with id 87847637c2bfb8270d08a8cc3850dfb053d0f2f1655dd9d1c6c5ff9631587458 Jul 22 16:30:30.854719 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:30.854679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qfnwc" event={"ID":"f9692eb5-5359-4ed0-9d66-6b93b03f1692","Type":"ContainerStarted","Data":"87847637c2bfb8270d08a8cc3850dfb053d0f2f1655dd9d1c6c5ff9631587458"} Jul 22 16:30:32.871724 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:32.871675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" event={"ID":"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c","Type":"ContainerStarted","Data":"bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3"} Jul 22 16:30:32.872164 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:32.871875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:32.910122 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:32.910052 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" podStartSLOduration=2.085636007 podStartE2EDuration="4.910033326s" totalImagesPullingTime="2.824397319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:30:32.907106047 +0000 UTC m=+590.487199964" watchObservedRunningTime="2026-07-22 16:30:32.910033326 +0000 UTC m=+590.490127222" Jul 22 16:30:34.886514 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:34.886476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qfnwc" event={"ID":"f9692eb5-5359-4ed0-9d66-6b93b03f1692","Type":"ContainerStarted","Data":"7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06"} Jul 22 16:30:34.903850 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:34.903790 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qfnwc" podStartSLOduration=1.958108625 podStartE2EDuration="5.903774347s" totalImagesPullingTime="3.945665722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:30:34.902708381 +0000 UTC m=+592.482802301" watchObservedRunningTime="2026-07-22 16:30:34.903774347 +0000 UTC m=+592.483868244" Jul 22 16:30:35.869671 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:35.869634 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qfnwc"] Jul 22 16:30:36.898923 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:36.898875 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qfnwc" podUID="f9692eb5-5359-4ed0-9d66-6b93b03f1692" containerName="authorino" containerID="cri-o://7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06" gracePeriod=30 Jul 22 16:30:37.158312 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.158226 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:37.287101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.287061 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsbrj\" (UniqueName: \"kubernetes.io/projected/f9692eb5-5359-4ed0-9d66-6b93b03f1692-kube-api-access-fsbrj\") pod \"f9692eb5-5359-4ed0-9d66-6b93b03f1692\" (UID: \"f9692eb5-5359-4ed0-9d66-6b93b03f1692\") " Jul 22 16:30:37.289381 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.289332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9692eb5-5359-4ed0-9d66-6b93b03f1692-kube-api-access-fsbrj" (OuterVolumeSpecName: "kube-api-access-fsbrj") pod "f9692eb5-5359-4ed0-9d66-6b93b03f1692" (UID: "f9692eb5-5359-4ed0-9d66-6b93b03f1692"). InnerVolumeSpecName "kube-api-access-fsbrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:30:37.388609 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.388560 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbrj\" (UniqueName: \"kubernetes.io/projected/f9692eb5-5359-4ed0-9d66-6b93b03f1692-kube-api-access-fsbrj\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:30:37.912824 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.912784 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9692eb5-5359-4ed0-9d66-6b93b03f1692" containerID="7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06" exitCode=0 Jul 22 16:30:37.913289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.912848 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qfnwc" Jul 22 16:30:37.913289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.912882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qfnwc" event={"ID":"f9692eb5-5359-4ed0-9d66-6b93b03f1692","Type":"ContainerDied","Data":"7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06"} Jul 22 16:30:37.913289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.912933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qfnwc" event={"ID":"f9692eb5-5359-4ed0-9d66-6b93b03f1692","Type":"ContainerDied","Data":"87847637c2bfb8270d08a8cc3850dfb053d0f2f1655dd9d1c6c5ff9631587458"} Jul 22 16:30:37.913289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.912953 2580 scope.go:117] "RemoveContainer" containerID="7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06" Jul 22 16:30:37.923098 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.923074 2580 scope.go:117] "RemoveContainer" containerID="7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06" Jul 22 16:30:37.923485 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:30:37.923461 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06\": container with ID starting with 7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06 not found: ID does not exist" containerID="7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06" Jul 22 16:30:37.923595 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.923492 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06"} err="failed to get container status \"7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06\": rpc error: code = NotFound desc = could not find container \"7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06\": container with ID starting with 7a1762c51824559c558394461cf375e78c4f27a4e1ee8d058d39a8695696fc06 not found: ID does not exist" Jul 22 16:30:37.937145 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.937107 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qfnwc"] Jul 22 16:30:37.943755 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:37.943714 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qfnwc"] Jul 22 16:30:38.979998 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:38.979960 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9692eb5-5359-4ed0-9d66-6b93b03f1692" path="/var/lib/kubelet/pods/f9692eb5-5359-4ed0-9d66-6b93b03f1692/volumes" Jul 22 16:30:43.880766 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:43.880731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:46.879644 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:46.879595 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:46.880110 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:46.879865 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" podUID="fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" containerName="limitador" containerID="cri-o://bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3" gracePeriod=30 Jul 22 16:30:47.446491 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.446385 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:47.586966 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.586921 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-config-file\") pod \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " Jul 22 16:30:47.587149 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.586982 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55brr\" (UniqueName: \"kubernetes.io/projected/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-kube-api-access-55brr\") pod \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\" (UID: \"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c\") " Jul 22 16:30:47.587294 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.587267 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-config-file" (OuterVolumeSpecName: "config-file") pod "fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" (UID: "fc9b51e0-94ae-4ec9-82c6-b60cb807da7c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:30:47.589202 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.589173 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-kube-api-access-55brr" (OuterVolumeSpecName: "kube-api-access-55brr") pod "fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" (UID: "fc9b51e0-94ae-4ec9-82c6-b60cb807da7c"). InnerVolumeSpecName "kube-api-access-55brr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:30:47.688561 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.688520 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-config-file\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:30:47.688561 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.688553 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55brr\" (UniqueName: \"kubernetes.io/projected/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c-kube-api-access-55brr\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:30:47.976490 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.976390 2580 generic.go:356] "Generic (PLEG): container finished" podID="fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" containerID="bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3" exitCode=0 Jul 22 16:30:47.976960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.976499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" event={"ID":"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c","Type":"ContainerDied","Data":"bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3"} Jul 22 16:30:47.976960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.976562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" event={"ID":"fc9b51e0-94ae-4ec9-82c6-b60cb807da7c","Type":"ContainerDied","Data":"a78c65e7d251861952cc5f2398a0152f2ddb8c0d41c0ce9bf37ac6f95afa7ee3"} Jul 22 16:30:47.976960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.976599 2580 scope.go:117] "RemoveContainer" containerID="bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3" Jul 22 16:30:47.976960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.976817 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fk6m4" Jul 22 16:30:47.990270 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.990243 2580 scope.go:117] "RemoveContainer" containerID="bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3" Jul 22 16:30:47.990643 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:30:47.990622 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3\": container with ID starting with bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3 not found: ID does not exist" containerID="bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3" Jul 22 16:30:47.990713 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:47.990653 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3"} err="failed to get container status \"bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3\": rpc error: code = NotFound desc = could not find container \"bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3\": container with ID starting with bffdefc11523f595a3ad598055f8aa4b336ea8d9a1cdf0511aeb4d6587fba6a3 not found: ID does not exist" Jul 22 16:30:48.001924 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:48.001890 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:48.006180 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:48.006154 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fk6m4"] Jul 22 16:30:48.980718 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:48.980686 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" path="/var/lib/kubelet/pods/fc9b51e0-94ae-4ec9-82c6-b60cb807da7c/volumes" Jul 22 16:30:50.595191 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595139 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj"] Jul 22 16:30:50.595812 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595792 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" containerName="limitador" Jul 22 16:30:50.595857 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595814 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" containerName="limitador" Jul 22 16:30:50.595857 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595846 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9692eb5-5359-4ed0-9d66-6b93b03f1692" containerName="authorino" Jul 22 16:30:50.595857 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595855 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9692eb5-5359-4ed0-9d66-6b93b03f1692" containerName="authorino" Jul 22 16:30:50.595968 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595956 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9b51e0-94ae-4ec9-82c6-b60cb807da7c" containerName="limitador" Jul 22 16:30:50.596008 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.595972 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9692eb5-5359-4ed0-9d66-6b93b03f1692" containerName="authorino" Jul 22 16:30:50.601078 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.601047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.603818 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.603790 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 16:30:50.604089 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.603849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9f9pv\"" Jul 22 16:30:50.604242 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.603861 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:30:50.604580 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.604563 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 16:30:50.605794 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.605764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj"] Jul 22 16:30:50.612008 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.611980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcszm\" (UniqueName: \"kubernetes.io/projected/4bf50400-f6a2-4e70-aa08-e70593892f30-kube-api-access-pcszm\") pod \"postgres-69fb4594cf-v9cwj\" (UID: \"4bf50400-f6a2-4e70-aa08-e70593892f30\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.612149 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.612084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4bf50400-f6a2-4e70-aa08-e70593892f30-data\") pod \"postgres-69fb4594cf-v9cwj\" (UID: \"4bf50400-f6a2-4e70-aa08-e70593892f30\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.712873 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.712827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcszm\" (UniqueName: \"kubernetes.io/projected/4bf50400-f6a2-4e70-aa08-e70593892f30-kube-api-access-pcszm\") pod \"postgres-69fb4594cf-v9cwj\" (UID: \"4bf50400-f6a2-4e70-aa08-e70593892f30\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.713065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.712909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4bf50400-f6a2-4e70-aa08-e70593892f30-data\") pod \"postgres-69fb4594cf-v9cwj\" (UID: \"4bf50400-f6a2-4e70-aa08-e70593892f30\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.713262 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.713247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4bf50400-f6a2-4e70-aa08-e70593892f30-data\") pod \"postgres-69fb4594cf-v9cwj\" (UID: \"4bf50400-f6a2-4e70-aa08-e70593892f30\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.721630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.721602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcszm\" (UniqueName: \"kubernetes.io/projected/4bf50400-f6a2-4e70-aa08-e70593892f30-kube-api-access-pcszm\") pod \"postgres-69fb4594cf-v9cwj\" (UID: \"4bf50400-f6a2-4e70-aa08-e70593892f30\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:50.914105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:50.913996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:51.060050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:51.060020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj"] Jul 22 16:30:51.062060 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:30:51.062034 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf50400_f6a2_4e70_aa08_e70593892f30.slice/crio-e5003112f380cac84df20a942fbb49c0acb164a7db53adfc80149fda39d6f208 WatchSource:0}: Error finding container e5003112f380cac84df20a942fbb49c0acb164a7db53adfc80149fda39d6f208: Status 404 returned error can't find the container with id e5003112f380cac84df20a942fbb49c0acb164a7db53adfc80149fda39d6f208 Jul 22 16:30:52.007707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:52.007664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" event={"ID":"4bf50400-f6a2-4e70-aa08-e70593892f30","Type":"ContainerStarted","Data":"e5003112f380cac84df20a942fbb49c0acb164a7db53adfc80149fda39d6f208"} Jul 22 16:30:56.037491 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:56.037443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" event={"ID":"4bf50400-f6a2-4e70-aa08-e70593892f30","Type":"ContainerStarted","Data":"d4557de18a3bbc0003dc19674b1a87defa6ebfb8c3db8ed05669812a269c1b79"} Jul 22 16:30:56.037920 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:56.037680 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:30:56.056705 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:30:56.056649 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" podStartSLOduration=1.187320441 podStartE2EDuration="6.056632584s" totalImagesPullingTime="4.869312143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:30:56.055532579 +0000 UTC m=+613.635626477" watchObservedRunningTime="2026-07-22 16:30:56.056632584 +0000 UTC m=+613.636726481" Jul 22 16:31:02.073826 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.073789 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v9cwj" Jul 22 16:31:02.679920 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.679882 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5dvlz"] Jul 22 16:31:02.686556 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.686534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:02.688995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.688966 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tfq9z\"" Jul 22 16:31:02.692148 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.692124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5dvlz"] Jul 22 16:31:02.731061 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.731019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhv7\" (UniqueName: \"kubernetes.io/projected/f9776ea7-eb28-4569-abc3-7af53747bb67-kube-api-access-fhhv7\") pod \"authorino-5d4b67c446-5dvlz\" (UID: \"f9776ea7-eb28-4569-abc3-7af53747bb67\") " pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:02.832081 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.832045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhv7\" (UniqueName: \"kubernetes.io/projected/f9776ea7-eb28-4569-abc3-7af53747bb67-kube-api-access-fhhv7\") pod \"authorino-5d4b67c446-5dvlz\" (UID: \"f9776ea7-eb28-4569-abc3-7af53747bb67\") " pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:02.840772 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.840738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhv7\" (UniqueName: \"kubernetes.io/projected/f9776ea7-eb28-4569-abc3-7af53747bb67-kube-api-access-fhhv7\") pod \"authorino-5d4b67c446-5dvlz\" (UID: \"f9776ea7-eb28-4569-abc3-7af53747bb67\") " pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:02.955022 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.954923 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5dvlz"] Jul 22 16:31:02.955208 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.955196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:02.981682 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.981655 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5855c98589-vh7nv"] Jul 22 16:31:02.987370 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.987345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:02.992547 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:02.992524 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5855c98589-vh7nv"] Jul 22 16:31:03.034579 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.034502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9lf\" (UniqueName: \"kubernetes.io/projected/79990214-8881-47fa-9caa-cbf94d09cf52-kube-api-access-pm9lf\") pod \"authorino-5855c98589-vh7nv\" (UID: \"79990214-8881-47fa-9caa-cbf94d09cf52\") " pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:03.105469 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.105439 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5dvlz"] Jul 22 16:31:03.107299 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:03.107259 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9776ea7_eb28_4569_abc3_7af53747bb67.slice/crio-63f946a02d00a76acf50ba8930de16e1531182c2b4136d3039505bc67af5d96e WatchSource:0}: Error finding container 63f946a02d00a76acf50ba8930de16e1531182c2b4136d3039505bc67af5d96e: Status 404 returned error can't find the container with id 63f946a02d00a76acf50ba8930de16e1531182c2b4136d3039505bc67af5d96e Jul 22 16:31:03.135117 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.135077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9lf\" (UniqueName: \"kubernetes.io/projected/79990214-8881-47fa-9caa-cbf94d09cf52-kube-api-access-pm9lf\") pod \"authorino-5855c98589-vh7nv\" (UID: \"79990214-8881-47fa-9caa-cbf94d09cf52\") " pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:03.145814 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.145767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9lf\" (UniqueName: \"kubernetes.io/projected/79990214-8881-47fa-9caa-cbf94d09cf52-kube-api-access-pm9lf\") pod \"authorino-5855c98589-vh7nv\" (UID: \"79990214-8881-47fa-9caa-cbf94d09cf52\") " pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:03.159718 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.159688 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5855c98589-vh7nv"] Jul 22 16:31:03.159986 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.159974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:03.189420 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.189367 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f69868c69-fkhkw"] Jul 22 16:31:03.194809 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.194786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.197857 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.197835 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 16:31:03.216380 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.212999 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f69868c69-fkhkw"] Jul 22 16:31:03.235633 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.235596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/548ff583-c4b3-4623-8d40-eb176437231a-tls-cert\") pod \"authorino-f69868c69-fkhkw\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.235764 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.235732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnst\" (UniqueName: \"kubernetes.io/projected/548ff583-c4b3-4623-8d40-eb176437231a-kube-api-access-vnnst\") pod \"authorino-f69868c69-fkhkw\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.304217 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.304188 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5855c98589-vh7nv"] Jul 22 16:31:03.306044 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:03.306019 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79990214_8881_47fa_9caa_cbf94d09cf52.slice/crio-04e38713a05f3595c621f05c12737a808a70e9d4b9c457322256b86503be3cf5 WatchSource:0}: Error finding container 04e38713a05f3595c621f05c12737a808a70e9d4b9c457322256b86503be3cf5: Status 404 returned error can't find the container with id 04e38713a05f3595c621f05c12737a808a70e9d4b9c457322256b86503be3cf5 Jul 22 16:31:03.337040 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.337002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnst\" (UniqueName: \"kubernetes.io/projected/548ff583-c4b3-4623-8d40-eb176437231a-kube-api-access-vnnst\") pod \"authorino-f69868c69-fkhkw\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.337226 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.337075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/548ff583-c4b3-4623-8d40-eb176437231a-tls-cert\") pod \"authorino-f69868c69-fkhkw\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.339530 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.339505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/548ff583-c4b3-4623-8d40-eb176437231a-tls-cert\") pod \"authorino-f69868c69-fkhkw\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.352299 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.352268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnst\" (UniqueName: \"kubernetes.io/projected/548ff583-c4b3-4623-8d40-eb176437231a-kube-api-access-vnnst\") pod \"authorino-f69868c69-fkhkw\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.507434 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.507406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:31:03.660965 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:03.660934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f69868c69-fkhkw"] Jul 22 16:31:03.662568 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:03.662541 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod548ff583_c4b3_4623_8d40_eb176437231a.slice/crio-428583565e756cd1f88a40fa8b24315bf920cbf76f05e831912ef1133c8f628d WatchSource:0}: Error finding container 428583565e756cd1f88a40fa8b24315bf920cbf76f05e831912ef1133c8f628d: Status 404 returned error can't find the container with id 428583565e756cd1f88a40fa8b24315bf920cbf76f05e831912ef1133c8f628d Jul 22 16:31:04.086261 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.086227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f69868c69-fkhkw" event={"ID":"548ff583-c4b3-4623-8d40-eb176437231a","Type":"ContainerStarted","Data":"428583565e756cd1f88a40fa8b24315bf920cbf76f05e831912ef1133c8f628d"} Jul 22 16:31:04.088185 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.088157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" event={"ID":"f9776ea7-eb28-4569-abc3-7af53747bb67","Type":"ContainerStarted","Data":"c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358"} Jul 22 16:31:04.088316 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.088203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" event={"ID":"f9776ea7-eb28-4569-abc3-7af53747bb67","Type":"ContainerStarted","Data":"63f946a02d00a76acf50ba8930de16e1531182c2b4136d3039505bc67af5d96e"} Jul 22 16:31:04.088316 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.088162 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" podUID="f9776ea7-eb28-4569-abc3-7af53747bb67" containerName="authorino" containerID="cri-o://c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358" gracePeriod=30 Jul 22 16:31:04.090015 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.089979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5855c98589-vh7nv" event={"ID":"79990214-8881-47fa-9caa-cbf94d09cf52","Type":"ContainerStarted","Data":"e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48"} Jul 22 16:31:04.090126 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.090031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5855c98589-vh7nv" event={"ID":"79990214-8881-47fa-9caa-cbf94d09cf52","Type":"ContainerStarted","Data":"04e38713a05f3595c621f05c12737a808a70e9d4b9c457322256b86503be3cf5"} Jul 22 16:31:04.090188 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.090137 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5855c98589-vh7nv" podUID="79990214-8881-47fa-9caa-cbf94d09cf52" containerName="authorino" containerID="cri-o://e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48" gracePeriod=30 Jul 22 16:31:04.107235 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.107190 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" podStartSLOduration=1.781047533 podStartE2EDuration="2.107171643s" totalImagesPullingTime="326.12411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:04.105888894 +0000 UTC m=+621.685982792" watchObservedRunningTime="2026-07-22 16:31:04.107171643 +0000 UTC m=+621.687265572" Jul 22 16:31:04.124409 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.124345 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5855c98589-vh7nv" podStartSLOduration=1.7339965149999998 podStartE2EDuration="2.124330414s" totalImagesPullingTime="390.333899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:04.122015641 +0000 UTC m=+621.702109559" watchObservedRunningTime="2026-07-22 16:31:04.124330414 +0000 UTC m=+621.704424323" Jul 22 16:31:04.366511 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.366489 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:04.372831 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.372809 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:04.449591 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.449553 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm9lf\" (UniqueName: \"kubernetes.io/projected/79990214-8881-47fa-9caa-cbf94d09cf52-kube-api-access-pm9lf\") pod \"79990214-8881-47fa-9caa-cbf94d09cf52\" (UID: \"79990214-8881-47fa-9caa-cbf94d09cf52\") " Jul 22 16:31:04.449770 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.449655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhhv7\" (UniqueName: \"kubernetes.io/projected/f9776ea7-eb28-4569-abc3-7af53747bb67-kube-api-access-fhhv7\") pod \"f9776ea7-eb28-4569-abc3-7af53747bb67\" (UID: \"f9776ea7-eb28-4569-abc3-7af53747bb67\") " Jul 22 16:31:04.451696 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.451671 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9776ea7-eb28-4569-abc3-7af53747bb67-kube-api-access-fhhv7" (OuterVolumeSpecName: "kube-api-access-fhhv7") pod "f9776ea7-eb28-4569-abc3-7af53747bb67" (UID: "f9776ea7-eb28-4569-abc3-7af53747bb67"). InnerVolumeSpecName "kube-api-access-fhhv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:31:04.451758 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.451721 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79990214-8881-47fa-9caa-cbf94d09cf52-kube-api-access-pm9lf" (OuterVolumeSpecName: "kube-api-access-pm9lf") pod "79990214-8881-47fa-9caa-cbf94d09cf52" (UID: "79990214-8881-47fa-9caa-cbf94d09cf52"). InnerVolumeSpecName "kube-api-access-pm9lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:31:04.550571 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.550533 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhhv7\" (UniqueName: \"kubernetes.io/projected/f9776ea7-eb28-4569-abc3-7af53747bb67-kube-api-access-fhhv7\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:31:04.550571 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:04.550562 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9lf\" (UniqueName: \"kubernetes.io/projected/79990214-8881-47fa-9caa-cbf94d09cf52-kube-api-access-pm9lf\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:31:05.097050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.096968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f69868c69-fkhkw" event={"ID":"548ff583-c4b3-4623-8d40-eb176437231a","Type":"ContainerStarted","Data":"4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1"} Jul 22 16:31:05.098281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.098257 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9776ea7-eb28-4569-abc3-7af53747bb67" containerID="c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358" exitCode=0 Jul 22 16:31:05.098377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.098291 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" Jul 22 16:31:05.098377 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.098339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" event={"ID":"f9776ea7-eb28-4569-abc3-7af53747bb67","Type":"ContainerDied","Data":"c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358"} Jul 22 16:31:05.098504 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.098378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5dvlz" event={"ID":"f9776ea7-eb28-4569-abc3-7af53747bb67","Type":"ContainerDied","Data":"63f946a02d00a76acf50ba8930de16e1531182c2b4136d3039505bc67af5d96e"} Jul 22 16:31:05.098504 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.098442 2580 scope.go:117] "RemoveContainer" containerID="c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358" Jul 22 16:31:05.099692 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.099614 2580 generic.go:356] "Generic (PLEG): container finished" podID="79990214-8881-47fa-9caa-cbf94d09cf52" containerID="e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48" exitCode=0 Jul 22 16:31:05.099692 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.099644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5855c98589-vh7nv" event={"ID":"79990214-8881-47fa-9caa-cbf94d09cf52","Type":"ContainerDied","Data":"e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48"} Jul 22 16:31:05.099692 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.099667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5855c98589-vh7nv" event={"ID":"79990214-8881-47fa-9caa-cbf94d09cf52","Type":"ContainerDied","Data":"04e38713a05f3595c621f05c12737a808a70e9d4b9c457322256b86503be3cf5"} Jul 22 16:31:05.099692 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.099647 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5855c98589-vh7nv" Jul 22 16:31:05.107817 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.107799 2580 scope.go:117] "RemoveContainer" containerID="c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358" Jul 22 16:31:05.108111 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:05.108089 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358\": container with ID starting with c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358 not found: ID does not exist" containerID="c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358" Jul 22 16:31:05.108161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.108115 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358"} err="failed to get container status \"c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358\": rpc error: code = NotFound desc = could not find container \"c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358\": container with ID starting with c5673fd8715391e7f8c878953118027aad2c304712a7ddc919d73d0526535358 not found: ID does not exist" Jul 22 16:31:05.108161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.108135 2580 scope.go:117] "RemoveContainer" containerID="e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48" Jul 22 16:31:05.115231 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.115192 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f69868c69-fkhkw" podStartSLOduration=1.703695551 podStartE2EDuration="2.115179215s" totalImagesPullingTime="411.483664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:05.113042457 +0000 UTC m=+622.693136350" watchObservedRunningTime="2026-07-22 16:31:05.115179215 +0000 UTC m=+622.695273111" Jul 22 16:31:05.116977 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.116954 2580 scope.go:117] "RemoveContainer" containerID="e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48" Jul 22 16:31:05.117226 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:05.117207 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48\": container with ID starting with e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48 not found: ID does not exist" containerID="e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48" Jul 22 16:31:05.117291 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.117236 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48"} err="failed to get container status \"e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48\": rpc error: code = NotFound desc = could not find container \"e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48\": container with ID starting with e2e60707a518b3bb3b644626354e2bdb96fa371da86888299e6dd7bc14480b48 not found: ID does not exist" Jul 22 16:31:05.130519 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.130491 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5dvlz"] Jul 22 16:31:05.134607 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.134576 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5dvlz"] Jul 22 16:31:05.149594 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.149563 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5855c98589-vh7nv"] Jul 22 16:31:05.156283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:05.156251 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5855c98589-vh7nv"] Jul 22 16:31:06.980464 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:06.980424 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79990214-8881-47fa-9caa-cbf94d09cf52" path="/var/lib/kubelet/pods/79990214-8881-47fa-9caa-cbf94d09cf52/volumes" Jul 22 16:31:06.981042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:06.981003 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9776ea7-eb28-4569-abc3-7af53747bb67" path="/var/lib/kubelet/pods/f9776ea7-eb28-4569-abc3-7af53747bb67/volumes" Jul 22 16:31:11.721463 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.721424 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77d9b4b94-jmkq4"] Jul 22 16:31:11.721922 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.721909 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9776ea7-eb28-4569-abc3-7af53747bb67" containerName="authorino" Jul 22 16:31:11.721967 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.721924 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9776ea7-eb28-4569-abc3-7af53747bb67" containerName="authorino" Jul 22 16:31:11.721967 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.721935 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79990214-8881-47fa-9caa-cbf94d09cf52" containerName="authorino" Jul 22 16:31:11.721967 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.721940 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79990214-8881-47fa-9caa-cbf94d09cf52" containerName="authorino" Jul 22 16:31:11.722074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.722023 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9776ea7-eb28-4569-abc3-7af53747bb67" containerName="authorino" Jul 22 16:31:11.722074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.722033 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79990214-8881-47fa-9caa-cbf94d09cf52" containerName="authorino" Jul 22 16:31:11.734956 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.734931 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77d9b4b94-jmkq4"] Jul 22 16:31:11.735115 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.735025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:11.737364 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.737337 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 16:31:11.737524 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.737345 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tf7g9\"" Jul 22 16:31:11.737524 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.737376 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 16:31:11.812626 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.812596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgffp\" (UniqueName: \"kubernetes.io/projected/4cf238c7-2317-4b21-bab6-6eb8468726c6-kube-api-access-tgffp\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:11.812846 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.812661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:11.913296 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.913260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:11.913501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.913329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgffp\" (UniqueName: \"kubernetes.io/projected/4cf238c7-2317-4b21-bab6-6eb8468726c6-kube-api-access-tgffp\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:11.913501 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:11.913439 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 22 16:31:11.913577 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:11.913512 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert podName:4cf238c7-2317-4b21-bab6-6eb8468726c6 nodeName:}" failed. No retries permitted until 2026-07-22 16:31:12.413494694 +0000 UTC m=+629.993588570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert") pod "maas-controller-77d9b4b94-jmkq4" (UID: "4cf238c7-2317-4b21-bab6-6eb8468726c6") : secret "maas-controller-webhook-cert" not found Jul 22 16:31:11.922726 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.921626 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76b7cff57-hwjcx"] Jul 22 16:31:11.926697 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.926671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgffp\" (UniqueName: \"kubernetes.io/projected/4cf238c7-2317-4b21-bab6-6eb8468726c6-kube-api-access-tgffp\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:11.927189 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.927170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:11.932706 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:11.932680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76b7cff57-hwjcx"] Jul 22 16:31:12.014832 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.014793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9dvm\" (UniqueName: \"kubernetes.io/projected/8f5f43d9-2743-4849-8100-2e73fcd2563b-kube-api-access-l9dvm\") pod \"maas-controller-76b7cff57-hwjcx\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.015030 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.015010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f5f43d9-2743-4849-8100-2e73fcd2563b-webhook-cert\") pod \"maas-controller-76b7cff57-hwjcx\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.115814 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.115784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f5f43d9-2743-4849-8100-2e73fcd2563b-webhook-cert\") pod \"maas-controller-76b7cff57-hwjcx\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.115997 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.115878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dvm\" (UniqueName: \"kubernetes.io/projected/8f5f43d9-2743-4849-8100-2e73fcd2563b-kube-api-access-l9dvm\") pod \"maas-controller-76b7cff57-hwjcx\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.118003 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.117980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f5f43d9-2743-4849-8100-2e73fcd2563b-webhook-cert\") pod \"maas-controller-76b7cff57-hwjcx\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.128083 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.128051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dvm\" (UniqueName: \"kubernetes.io/projected/8f5f43d9-2743-4849-8100-2e73fcd2563b-kube-api-access-l9dvm\") pod \"maas-controller-76b7cff57-hwjcx\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.248838 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.248801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:12.385076 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.385043 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76b7cff57-hwjcx"] Jul 22 16:31:12.386915 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:12.386886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5f43d9_2743_4849_8100_2e73fcd2563b.slice/crio-e9e256a9e187b8abe7b3bf5bcf3c224e10ccbb317d37f35636086f6fc89d4ef8 WatchSource:0}: Error finding container e9e256a9e187b8abe7b3bf5bcf3c224e10ccbb317d37f35636086f6fc89d4ef8: Status 404 returned error can't find the container with id e9e256a9e187b8abe7b3bf5bcf3c224e10ccbb317d37f35636086f6fc89d4ef8 Jul 22 16:31:12.418944 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.418906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:12.421349 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.421323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert\") pod \"maas-controller-77d9b4b94-jmkq4\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:12.645927 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.645844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:12.804083 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:12.804055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77d9b4b94-jmkq4"] Jul 22 16:31:12.806312 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:12.806283 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf238c7_2317_4b21_bab6_6eb8468726c6.slice/crio-5e7b764f76a116f519bde3e92039533256c8f4239c4e30c57b9cb629093cb4fd WatchSource:0}: Error finding container 5e7b764f76a116f519bde3e92039533256c8f4239c4e30c57b9cb629093cb4fd: Status 404 returned error can't find the container with id 5e7b764f76a116f519bde3e92039533256c8f4239c4e30c57b9cb629093cb4fd Jul 22 16:31:13.150552 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:13.150509 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" event={"ID":"4cf238c7-2317-4b21-bab6-6eb8468726c6","Type":"ContainerStarted","Data":"5e7b764f76a116f519bde3e92039533256c8f4239c4e30c57b9cb629093cb4fd"} Jul 22 16:31:13.151938 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:13.151901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76b7cff57-hwjcx" event={"ID":"8f5f43d9-2743-4849-8100-2e73fcd2563b","Type":"ContainerStarted","Data":"e9e256a9e187b8abe7b3bf5bcf3c224e10ccbb317d37f35636086f6fc89d4ef8"} Jul 22 16:31:16.173330 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:16.173293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76b7cff57-hwjcx" event={"ID":"8f5f43d9-2743-4849-8100-2e73fcd2563b","Type":"ContainerStarted","Data":"92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f"} Jul 22 16:31:16.173801 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:16.173405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:16.174848 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:16.174820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" event={"ID":"4cf238c7-2317-4b21-bab6-6eb8468726c6","Type":"ContainerStarted","Data":"30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c"} Jul 22 16:31:16.174968 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:16.174915 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:16.199267 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:16.199216 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76b7cff57-hwjcx" podStartSLOduration=2.382465452 podStartE2EDuration="5.199199023s" totalImagesPullingTime="2.816733571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:16.197835933 +0000 UTC m=+633.777929830" watchObservedRunningTime="2026-07-22 16:31:16.199199023 +0000 UTC m=+633.779292929" Jul 22 16:31:16.216137 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:16.216084 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" podStartSLOduration=2.8122952359999998 podStartE2EDuration="5.216063581s" totalImagesPullingTime="2.403768345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:16.215817168 +0000 UTC m=+633.795911118" watchObservedRunningTime="2026-07-22 16:31:16.216063581 +0000 UTC m=+633.796157480" Jul 22 16:31:27.183225 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.183187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:31:27.183705 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.183440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:27.239219 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.239178 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77d9b4b94-jmkq4"] Jul 22 16:31:27.239464 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.239419 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" podUID="4cf238c7-2317-4b21-bab6-6eb8468726c6" containerName="manager" containerID="cri-o://30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c" gracePeriod=30 Jul 22 16:31:27.490908 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.490873 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:27.561311 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.561278 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert\") pod \"4cf238c7-2317-4b21-bab6-6eb8468726c6\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " Jul 22 16:31:27.561537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.561332 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgffp\" (UniqueName: \"kubernetes.io/projected/4cf238c7-2317-4b21-bab6-6eb8468726c6-kube-api-access-tgffp\") pod \"4cf238c7-2317-4b21-bab6-6eb8468726c6\" (UID: \"4cf238c7-2317-4b21-bab6-6eb8468726c6\") " Jul 22 16:31:27.563636 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.563601 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4cf238c7-2317-4b21-bab6-6eb8468726c6" (UID: "4cf238c7-2317-4b21-bab6-6eb8468726c6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:31:27.563757 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.563640 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf238c7-2317-4b21-bab6-6eb8468726c6-kube-api-access-tgffp" (OuterVolumeSpecName: "kube-api-access-tgffp") pod "4cf238c7-2317-4b21-bab6-6eb8468726c6" (UID: "4cf238c7-2317-4b21-bab6-6eb8468726c6"). InnerVolumeSpecName "kube-api-access-tgffp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:31:27.662417 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.662344 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cf238c7-2317-4b21-bab6-6eb8468726c6-webhook-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:31:27.662417 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:27.662387 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgffp\" (UniqueName: \"kubernetes.io/projected/4cf238c7-2317-4b21-bab6-6eb8468726c6-kube-api-access-tgffp\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:31:28.244704 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.244662 2580 generic.go:356] "Generic (PLEG): container finished" podID="4cf238c7-2317-4b21-bab6-6eb8468726c6" containerID="30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c" exitCode=0 Jul 22 16:31:28.245105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.244710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" event={"ID":"4cf238c7-2317-4b21-bab6-6eb8468726c6","Type":"ContainerDied","Data":"30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c"} Jul 22 16:31:28.245105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.244721 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" Jul 22 16:31:28.245105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.244764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d9b4b94-jmkq4" event={"ID":"4cf238c7-2317-4b21-bab6-6eb8468726c6","Type":"ContainerDied","Data":"5e7b764f76a116f519bde3e92039533256c8f4239c4e30c57b9cb629093cb4fd"} Jul 22 16:31:28.245105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.244795 2580 scope.go:117] "RemoveContainer" containerID="30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c" Jul 22 16:31:28.254583 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.254562 2580 scope.go:117] "RemoveContainer" containerID="30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c" Jul 22 16:31:28.254836 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:28.254815 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c\": container with ID starting with 30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c not found: ID does not exist" containerID="30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c" Jul 22 16:31:28.254885 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.254845 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c"} err="failed to get container status \"30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c\": rpc error: code = NotFound desc = could not find container \"30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c\": container with ID starting with 30e63df4368b07990cee4673b669f5bd823e02f2054f3ceba4f783ee30a2d41c not found: ID does not exist" Jul 22 16:31:28.267556 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.267531 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77d9b4b94-jmkq4"] Jul 22 16:31:28.271888 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.271864 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77d9b4b94-jmkq4"] Jul 22 16:31:28.979746 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:28.979713 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf238c7-2317-4b21-bab6-6eb8468726c6" path="/var/lib/kubelet/pods/4cf238c7-2317-4b21-bab6-6eb8468726c6/volumes" Jul 22 16:31:47.584586 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.584554 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh"] Jul 22 16:31:47.584982 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.584970 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf238c7-2317-4b21-bab6-6eb8468726c6" containerName="manager" Jul 22 16:31:47.585027 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.584983 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf238c7-2317-4b21-bab6-6eb8468726c6" containerName="manager" Jul 22 16:31:47.585078 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.585069 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf238c7-2317-4b21-bab6-6eb8468726c6" containerName="manager" Jul 22 16:31:47.594207 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.594178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.596585 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.596556 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 16:31:47.596726 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.596625 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwhb6\"" Jul 22 16:31:47.601188 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.601164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh"] Jul 22 16:31:47.644674 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.644637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhnb\" (UniqueName: \"kubernetes.io/projected/c769ec4e-3aad-44ef-a2aa-974de95c749d-kube-api-access-swhnb\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.644674 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.644676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769ec4e-3aad-44ef-a2aa-974de95c749d-openshift-service-ca\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.644931 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.644779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769ec4e-3aad-44ef-a2aa-974de95c749d-maas-api-tls\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.745657 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.745619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhnb\" (UniqueName: \"kubernetes.io/projected/c769ec4e-3aad-44ef-a2aa-974de95c749d-kube-api-access-swhnb\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.745834 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.745670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769ec4e-3aad-44ef-a2aa-974de95c749d-openshift-service-ca\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.745834 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.745738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769ec4e-3aad-44ef-a2aa-974de95c749d-maas-api-tls\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.745927 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:47.745903 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 22 16:31:47.745989 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:31:47.745977 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c769ec4e-3aad-44ef-a2aa-974de95c749d-maas-api-tls podName:c769ec4e-3aad-44ef-a2aa-974de95c749d nodeName:}" failed. No retries permitted until 2026-07-22 16:31:48.245957636 +0000 UTC m=+665.826051515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c769ec4e-3aad-44ef-a2aa-974de95c749d-maas-api-tls") pod "maas-api-d99d56b6d-zmmkh" (UID: "c769ec4e-3aad-44ef-a2aa-974de95c749d") : secret "maas-api-serving-cert" not found Jul 22 16:31:47.746419 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.746383 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769ec4e-3aad-44ef-a2aa-974de95c749d-openshift-service-ca\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:47.754784 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:47.754762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhnb\" (UniqueName: \"kubernetes.io/projected/c769ec4e-3aad-44ef-a2aa-974de95c749d-kube-api-access-swhnb\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:48.155699 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.155658 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xtpcr"] Jul 22 16:31:48.160061 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.160035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.162488 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.162465 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9wf4v\"" Jul 22 16:31:48.162610 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.162473 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 16:31:48.170246 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.170216 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xtpcr"] Jul 22 16:31:48.248925 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.248892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769ec4e-3aad-44ef-a2aa-974de95c749d-maas-api-tls\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:48.249103 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.248946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92d4bce-d55a-4967-a42e-62d9f8bd7fae-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xtpcr\" (UID: \"a92d4bce-d55a-4967-a42e-62d9f8bd7fae\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.249103 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.249024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbv2x\" (UniqueName: \"kubernetes.io/projected/a92d4bce-d55a-4967-a42e-62d9f8bd7fae-kube-api-access-fbv2x\") pod \"payload-pre-processing-7677b774bb-xtpcr\" (UID: \"a92d4bce-d55a-4967-a42e-62d9f8bd7fae\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.251518 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.251493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769ec4e-3aad-44ef-a2aa-974de95c749d-maas-api-tls\") pod \"maas-api-d99d56b6d-zmmkh\" (UID: \"c769ec4e-3aad-44ef-a2aa-974de95c749d\") " pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:48.349510 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.349475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbv2x\" (UniqueName: \"kubernetes.io/projected/a92d4bce-d55a-4967-a42e-62d9f8bd7fae-kube-api-access-fbv2x\") pod \"payload-pre-processing-7677b774bb-xtpcr\" (UID: \"a92d4bce-d55a-4967-a42e-62d9f8bd7fae\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.349663 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.349577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92d4bce-d55a-4967-a42e-62d9f8bd7fae-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xtpcr\" (UID: \"a92d4bce-d55a-4967-a42e-62d9f8bd7fae\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.350269 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.350250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92d4bce-d55a-4967-a42e-62d9f8bd7fae-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xtpcr\" (UID: \"a92d4bce-d55a-4967-a42e-62d9f8bd7fae\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.360040 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.360010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbv2x\" (UniqueName: \"kubernetes.io/projected/a92d4bce-d55a-4967-a42e-62d9f8bd7fae-kube-api-access-fbv2x\") pod \"payload-pre-processing-7677b774bb-xtpcr\" (UID: \"a92d4bce-d55a-4967-a42e-62d9f8bd7fae\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.472667 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.472574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:48.505333 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.505304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:48.632733 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.632696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xtpcr"] Jul 22 16:31:48.635061 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:48.635028 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92d4bce_d55a_4967_a42e_62d9f8bd7fae.slice/crio-ab75f1dfc54d5810b996c3d6ce1b31e67a338719b1ed76880884c594915434e8 WatchSource:0}: Error finding container ab75f1dfc54d5810b996c3d6ce1b31e67a338719b1ed76880884c594915434e8: Status 404 returned error can't find the container with id ab75f1dfc54d5810b996c3d6ce1b31e67a338719b1ed76880884c594915434e8 Jul 22 16:31:48.637236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.637219 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:31:48.661849 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:48.661822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh"] Jul 22 16:31:48.663379 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:31:48.663351 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc769ec4e_3aad_44ef_a2aa_974de95c749d.slice/crio-debf03dde2b05ced9449bdbb3dacc143a066a2c932c3b4389f0efc034386da08 WatchSource:0}: Error finding container debf03dde2b05ced9449bdbb3dacc143a066a2c932c3b4389f0efc034386da08: Status 404 returned error can't find the container with id debf03dde2b05ced9449bdbb3dacc143a066a2c932c3b4389f0efc034386da08 Jul 22 16:31:49.378751 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:49.378704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" event={"ID":"c769ec4e-3aad-44ef-a2aa-974de95c749d","Type":"ContainerStarted","Data":"debf03dde2b05ced9449bdbb3dacc143a066a2c932c3b4389f0efc034386da08"} Jul 22 16:31:49.381236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:49.381203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" event={"ID":"a92d4bce-d55a-4967-a42e-62d9f8bd7fae","Type":"ContainerStarted","Data":"ab75f1dfc54d5810b996c3d6ce1b31e67a338719b1ed76880884c594915434e8"} Jul 22 16:31:51.395861 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:51.395816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" event={"ID":"a92d4bce-d55a-4967-a42e-62d9f8bd7fae","Type":"ContainerStarted","Data":"5063756a9ba029a260f831229fadca3f7d7e52345c4f7335f78b48bbc1668578"} Jul 22 16:31:51.396277 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:51.395972 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:31:51.397261 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:51.397238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" event={"ID":"c769ec4e-3aad-44ef-a2aa-974de95c749d","Type":"ContainerStarted","Data":"23787f06b2b49ae14da99bc9445ee29f4a883e6eda160bcd592984e81c112b78"} Jul 22 16:31:51.397337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:51.397295 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:31:51.414791 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:51.414752 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" podStartSLOduration=1.525447628 podStartE2EDuration="3.4147409s" totalImagesPullingTime="1.889293272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:51.412322126 +0000 UTC m=+668.992416025" watchObservedRunningTime="2026-07-22 16:31:51.4147409 +0000 UTC m=+668.994834797" Jul 22 16:31:51.432274 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:51.432234 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" podStartSLOduration=2.570393041 podStartE2EDuration="4.432222731s" totalImagesPullingTime="1.86182969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:51.429217019 +0000 UTC m=+669.009310945" watchObservedRunningTime="2026-07-22 16:31:51.432222731 +0000 UTC m=+669.012316627" Jul 22 16:31:57.408448 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:31:57.408412 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d99d56b6d-zmmkh" Jul 22 16:32:02.405035 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:02.404995 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" podUID="a92d4bce-d55a-4967-a42e-62d9f8bd7fae" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.60:9004: connect: connection refused" Jul 22 16:32:07.488584 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.488550 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8"] Jul 22 16:32:07.493454 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.493426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.495966 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.495941 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 16:32:07.496772 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.496747 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 16:32:07.496933 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.496747 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b2274\"" Jul 22 16:32:07.496933 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.496815 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 16:32:07.504272 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.504240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8"] Jul 22 16:32:07.632991 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.632951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.633205 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.633026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.633205 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.633057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.633205 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.633111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgst\" (UniqueName: \"kubernetes.io/projected/6525da05-8827-4a3a-a267-a21d83b370c7-kube-api-access-fmgst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.633205 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.633157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.633205 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.633189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6525da05-8827-4a3a-a267-a21d83b370c7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgst\" (UniqueName: \"kubernetes.io/projected/6525da05-8827-4a3a-a267-a21d83b370c7-kube-api-access-fmgst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734316 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734381 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6525da05-8827-4a3a-a267-a21d83b370c7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734482 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734590 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.734906 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.735013 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.734989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.735161 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.735134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.736545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.736521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6525da05-8827-4a3a-a267-a21d83b370c7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.737089 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.737072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6525da05-8827-4a3a-a267-a21d83b370c7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.742960 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.742904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgst\" (UniqueName: \"kubernetes.io/projected/6525da05-8827-4a3a-a267-a21d83b370c7-kube-api-access-fmgst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8\" (UID: \"6525da05-8827-4a3a-a267-a21d83b370c7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.806614 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.806572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:07.977118 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:07.977085 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8"] Jul 22 16:32:07.977982 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:32:07.977944 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6525da05_8827_4a3a_a267_a21d83b370c7.slice/crio-d31082d79fd4a322e475e8ba1d8ff380620db37d6df9c23b98ecb6f6787b4b97 WatchSource:0}: Error finding container d31082d79fd4a322e475e8ba1d8ff380620db37d6df9c23b98ecb6f6787b4b97: Status 404 returned error can't find the container with id d31082d79fd4a322e475e8ba1d8ff380620db37d6df9c23b98ecb6f6787b4b97 Jul 22 16:32:08.473774 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:08.473727 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" podUID="a92d4bce-d55a-4967-a42e-62d9f8bd7fae" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.60:9004: connect: connection refused" Jul 22 16:32:08.500200 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:08.500155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" event={"ID":"6525da05-8827-4a3a-a267-a21d83b370c7","Type":"ContainerStarted","Data":"d31082d79fd4a322e475e8ba1d8ff380620db37d6df9c23b98ecb6f6787b4b97"} Jul 22 16:32:09.507607 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:09.507567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" event={"ID":"6525da05-8827-4a3a-a267-a21d83b370c7","Type":"ContainerStarted","Data":"71e16a47f996425b9a0d5b2928936925f7376c62f85fddd3fa1f8d9fcfcd6a1a"} Jul 22 16:32:09.530711 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:09.530659 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" podStartSLOduration=1.213194624 podStartE2EDuration="2.530641814s" totalImagesPullingTime="1.31744719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:32:09.528725128 +0000 UTC m=+687.108819046" watchObservedRunningTime="2026-07-22 16:32:09.530641814 +0000 UTC m=+687.110735710" Jul 22 16:32:12.404506 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:12.404469 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xtpcr" Jul 22 16:32:17.807147 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:17.807102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:17.807578 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:17.807179 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:17.820995 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:17.820968 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:18.578728 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:18.578701 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8" Jul 22 16:32:39.093465 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.093374 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d"] Jul 22 16:32:39.097108 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.097091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.099573 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.099553 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 16:32:39.106715 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.106692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d"] Jul 22 16:32:39.149635 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.149602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a64d32-758d-4940-87e3-2887ce4c70c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.149822 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.149645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.149822 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.149664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595mb\" (UniqueName: \"kubernetes.io/projected/30a64d32-758d-4940-87e3-2887ce4c70c8-kube-api-access-595mb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.149822 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.149685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.149993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.149830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.149993 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.149886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251033 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.250990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a64d32-758d-4940-87e3-2887ce4c70c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595mb\" (UniqueName: \"kubernetes.io/projected/30a64d32-758d-4940-87e3-2887ce4c70c8-kube-api-access-595mb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251606 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251649 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.251649 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.251636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.253407 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.253375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a64d32-758d-4940-87e3-2887ce4c70c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.253741 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.253721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a64d32-758d-4940-87e3-2887ce4c70c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.260031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.260008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595mb\" (UniqueName: \"kubernetes.io/projected/30a64d32-758d-4940-87e3-2887ce4c70c8-kube-api-access-595mb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d\" (UID: \"30a64d32-758d-4940-87e3-2887ce4c70c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.408852 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.408766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:39.562214 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.562186 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d"] Jul 22 16:32:39.563434 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:32:39.563382 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a64d32_758d_4940_87e3_2887ce4c70c8.slice/crio-57671fe09132d6e3379f6b5911b0badcabc92158ae5763c5e57b58a5708313a8 WatchSource:0}: Error finding container 57671fe09132d6e3379f6b5911b0badcabc92158ae5763c5e57b58a5708313a8: Status 404 returned error can't find the container with id 57671fe09132d6e3379f6b5911b0badcabc92158ae5763c5e57b58a5708313a8 Jul 22 16:32:39.688487 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:39.688376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" event={"ID":"30a64d32-758d-4940-87e3-2887ce4c70c8","Type":"ContainerStarted","Data":"57671fe09132d6e3379f6b5911b0badcabc92158ae5763c5e57b58a5708313a8"} Jul 22 16:32:40.697550 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:40.697512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" event={"ID":"30a64d32-758d-4940-87e3-2887ce4c70c8","Type":"ContainerStarted","Data":"ac9080f1582fc4ff23a7d8c9e85c28630621281daf7326714b2beba9d4fd9ea6"} Jul 22 16:32:40.718654 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:40.718598 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" podStartSLOduration=1.4899988259999999 podStartE2EDuration="1.718582323s" totalImagesPullingTime="228.583497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:32:40.716732489 +0000 UTC m=+718.296826386" watchObservedRunningTime="2026-07-22 16:32:40.718582323 +0000 UTC m=+718.298676219" Jul 22 16:32:49.409636 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:49.409598 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:49.410092 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:49.409658 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:49.421850 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:49.421828 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:32:49.763424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:32:49.763376 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d" Jul 22 16:34:29.413754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.413723 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64b8879888-w6k2j"] Jul 22 16:34:29.417473 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.417456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.423839 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.423809 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b8879888-w6k2j"] Jul 22 16:34:29.533630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.533586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02c3c4b6-1235-455a-bfb4-d38e9e95eb6e-tls-cert\") pod \"authorino-64b8879888-w6k2j\" (UID: \"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e\") " pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.533812 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.533640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rdv\" (UniqueName: \"kubernetes.io/projected/02c3c4b6-1235-455a-bfb4-d38e9e95eb6e-kube-api-access-25rdv\") pod \"authorino-64b8879888-w6k2j\" (UID: \"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e\") " pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.634661 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.634616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rdv\" (UniqueName: \"kubernetes.io/projected/02c3c4b6-1235-455a-bfb4-d38e9e95eb6e-kube-api-access-25rdv\") pod \"authorino-64b8879888-w6k2j\" (UID: \"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e\") " pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.634864 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.634792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02c3c4b6-1235-455a-bfb4-d38e9e95eb6e-tls-cert\") pod \"authorino-64b8879888-w6k2j\" (UID: \"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e\") " pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.637184 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.637158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02c3c4b6-1235-455a-bfb4-d38e9e95eb6e-tls-cert\") pod \"authorino-64b8879888-w6k2j\" (UID: \"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e\") " pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.643072 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.643038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rdv\" (UniqueName: \"kubernetes.io/projected/02c3c4b6-1235-455a-bfb4-d38e9e95eb6e-kube-api-access-25rdv\") pod \"authorino-64b8879888-w6k2j\" (UID: \"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e\") " pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.728689 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.728606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b8879888-w6k2j" Jul 22 16:34:29.897947 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:29.897917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b8879888-w6k2j"] Jul 22 16:34:29.899583 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:34:29.899555 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c3c4b6_1235_455a_bfb4_d38e9e95eb6e.slice/crio-fd915fc09eb236ed86bb7d26b0e0d00dc60699810a128d5b7b71a03572975971 WatchSource:0}: Error finding container fd915fc09eb236ed86bb7d26b0e0d00dc60699810a128d5b7b71a03572975971: Status 404 returned error can't find the container with id fd915fc09eb236ed86bb7d26b0e0d00dc60699810a128d5b7b71a03572975971 Jul 22 16:34:30.353941 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.353902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b8879888-w6k2j" event={"ID":"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e","Type":"ContainerStarted","Data":"f3082e8cdae53eb3760c45b7d18b1e01b94a5b5db7dfad27ef0e38f022fc931e"} Jul 22 16:34:30.354168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.353955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b8879888-w6k2j" event={"ID":"02c3c4b6-1235-455a-bfb4-d38e9e95eb6e","Type":"ContainerStarted","Data":"fd915fc09eb236ed86bb7d26b0e0d00dc60699810a128d5b7b71a03572975971"} Jul 22 16:34:30.370957 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.370907 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64b8879888-w6k2j" podStartSLOduration=0.999038202 podStartE2EDuration="1.370890573s" totalImagesPullingTime="371.852371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:34:30.370058714 +0000 UTC m=+827.950152627" watchObservedRunningTime="2026-07-22 16:34:30.370890573 +0000 UTC m=+827.950984470" Jul 22 16:34:30.398062 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.398029 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f69868c69-fkhkw"] Jul 22 16:34:30.398261 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.398237 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f69868c69-fkhkw" podUID="548ff583-c4b3-4623-8d40-eb176437231a" containerName="authorino" containerID="cri-o://4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1" gracePeriod=30 Jul 22 16:34:30.679494 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.679463 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:34:30.746074 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.746042 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/548ff583-c4b3-4623-8d40-eb176437231a-tls-cert\") pod \"548ff583-c4b3-4623-8d40-eb176437231a\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " Jul 22 16:34:30.746241 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.746175 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnnst\" (UniqueName: \"kubernetes.io/projected/548ff583-c4b3-4623-8d40-eb176437231a-kube-api-access-vnnst\") pod \"548ff583-c4b3-4623-8d40-eb176437231a\" (UID: \"548ff583-c4b3-4623-8d40-eb176437231a\") " Jul 22 16:34:30.748267 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.748227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548ff583-c4b3-4623-8d40-eb176437231a-kube-api-access-vnnst" (OuterVolumeSpecName: "kube-api-access-vnnst") pod "548ff583-c4b3-4623-8d40-eb176437231a" (UID: "548ff583-c4b3-4623-8d40-eb176437231a"). InnerVolumeSpecName "kube-api-access-vnnst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:34:30.758505 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.758467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548ff583-c4b3-4623-8d40-eb176437231a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "548ff583-c4b3-4623-8d40-eb176437231a" (UID: "548ff583-c4b3-4623-8d40-eb176437231a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:34:30.847149 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.847062 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnst\" (UniqueName: \"kubernetes.io/projected/548ff583-c4b3-4623-8d40-eb176437231a-kube-api-access-vnnst\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:34:30.847149 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:30.847094 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/548ff583-c4b3-4623-8d40-eb176437231a-tls-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:34:31.362335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.362297 2580 generic.go:356] "Generic (PLEG): container finished" podID="548ff583-c4b3-4623-8d40-eb176437231a" containerID="4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1" exitCode=0 Jul 22 16:34:31.362537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.362341 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f69868c69-fkhkw" Jul 22 16:34:31.362537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.362383 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f69868c69-fkhkw" event={"ID":"548ff583-c4b3-4623-8d40-eb176437231a","Type":"ContainerDied","Data":"4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1"} Jul 22 16:34:31.362537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.362443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f69868c69-fkhkw" event={"ID":"548ff583-c4b3-4623-8d40-eb176437231a","Type":"ContainerDied","Data":"428583565e756cd1f88a40fa8b24315bf920cbf76f05e831912ef1133c8f628d"} Jul 22 16:34:31.362537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.362465 2580 scope.go:117] "RemoveContainer" containerID="4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1" Jul 22 16:34:31.374335 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.374305 2580 scope.go:117] "RemoveContainer" containerID="4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1" Jul 22 16:34:31.374689 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:34:31.374666 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1\": container with ID starting with 4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1 not found: ID does not exist" containerID="4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1" Jul 22 16:34:31.374754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.374699 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1"} err="failed to get container status \"4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1\": rpc error: code = NotFound desc = could not find container \"4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1\": container with ID starting with 4523d5c34204f95df51c04b1d05ae9dde4fe81804458e8ccadcbeb0212e10fa1 not found: ID does not exist" Jul 22 16:34:31.380191 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.380159 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f69868c69-fkhkw"] Jul 22 16:34:31.386209 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:31.386175 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f69868c69-fkhkw"] Jul 22 16:34:32.980712 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:34:32.980678 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548ff583-c4b3-4623-8d40-eb176437231a" path="/var/lib/kubelet/pods/548ff583-c4b3-4623-8d40-eb176437231a/volumes" Jul 22 16:35:19.077084 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.077048 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76b7cff57-hwjcx"] Jul 22 16:35:19.077625 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.077270 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76b7cff57-hwjcx" podUID="8f5f43d9-2743-4849-8100-2e73fcd2563b" containerName="manager" containerID="cri-o://92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f" gracePeriod=30 Jul 22 16:35:19.356251 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.356221 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:35:19.451936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.451878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9dvm\" (UniqueName: \"kubernetes.io/projected/8f5f43d9-2743-4849-8100-2e73fcd2563b-kube-api-access-l9dvm\") pod \"8f5f43d9-2743-4849-8100-2e73fcd2563b\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " Jul 22 16:35:19.452157 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.452010 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f5f43d9-2743-4849-8100-2e73fcd2563b-webhook-cert\") pod \"8f5f43d9-2743-4849-8100-2e73fcd2563b\" (UID: \"8f5f43d9-2743-4849-8100-2e73fcd2563b\") " Jul 22 16:35:19.456463 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.455505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5f43d9-2743-4849-8100-2e73fcd2563b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8f5f43d9-2743-4849-8100-2e73fcd2563b" (UID: "8f5f43d9-2743-4849-8100-2e73fcd2563b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:35:19.462527 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.462500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5f43d9-2743-4849-8100-2e73fcd2563b-kube-api-access-l9dvm" (OuterVolumeSpecName: "kube-api-access-l9dvm") pod "8f5f43d9-2743-4849-8100-2e73fcd2563b" (UID: "8f5f43d9-2743-4849-8100-2e73fcd2563b"). InnerVolumeSpecName "kube-api-access-l9dvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:35:19.553881 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.553846 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9dvm\" (UniqueName: \"kubernetes.io/projected/8f5f43d9-2743-4849-8100-2e73fcd2563b-kube-api-access-l9dvm\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:35:19.553881 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.553874 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f5f43d9-2743-4849-8100-2e73fcd2563b-webhook-cert\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:35:19.667482 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.667381 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f5f43d9-2743-4849-8100-2e73fcd2563b" containerID="92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f" exitCode=0 Jul 22 16:35:19.667482 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.667428 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76b7cff57-hwjcx" event={"ID":"8f5f43d9-2743-4849-8100-2e73fcd2563b","Type":"ContainerDied","Data":"92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f"} Jul 22 16:35:19.667659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.667483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76b7cff57-hwjcx" event={"ID":"8f5f43d9-2743-4849-8100-2e73fcd2563b","Type":"ContainerDied","Data":"e9e256a9e187b8abe7b3bf5bcf3c224e10ccbb317d37f35636086f6fc89d4ef8"} Jul 22 16:35:19.667659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.667443 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76b7cff57-hwjcx" Jul 22 16:35:19.667659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.667509 2580 scope.go:117] "RemoveContainer" containerID="92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f" Jul 22 16:35:19.677577 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.677560 2580 scope.go:117] "RemoveContainer" containerID="92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f" Jul 22 16:35:19.677858 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:35:19.677837 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f\": container with ID starting with 92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f not found: ID does not exist" containerID="92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f" Jul 22 16:35:19.677908 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.677869 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f"} err="failed to get container status \"92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f\": rpc error: code = NotFound desc = could not find container \"92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f\": container with ID starting with 92922320abbf008e3b093665c71bad44749213527de09dc31fe9b3be85fd797f not found: ID does not exist" Jul 22 16:35:19.692445 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.692421 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76b7cff57-hwjcx"] Jul 22 16:35:19.694836 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:19.694816 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76b7cff57-hwjcx"] Jul 22 16:35:20.979822 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:35:20.979790 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f5f43d9-2743-4849-8100-2e73fcd2563b" path="/var/lib/kubelet/pods/8f5f43d9-2743-4849-8100-2e73fcd2563b/volumes" Jul 22 16:41:27.970545 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:27.970509 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm"] Jul 22 16:41:27.973070 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:27.970745 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" podUID="88c8d142-3b71-4623-be8a-d258bbfda6ca" containerName="manager" containerID="cri-o://5e94225f666ef86558e97a2b0970be24ca159053d0e65661a98bdc01e38ad31d" gracePeriod=10 Jul 22 16:41:29.661734 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:29.661680 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" podUID="88c8d142-3b71-4623-be8a-d258bbfda6ca" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.50:8081/readyz\": dial tcp 10.133.0.50:8081: connect: connection refused" Jul 22 16:41:29.856529 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:29.856499 2580 generic.go:356] "Generic (PLEG): container finished" podID="88c8d142-3b71-4623-be8a-d258bbfda6ca" containerID="5e94225f666ef86558e97a2b0970be24ca159053d0e65661a98bdc01e38ad31d" exitCode=0 Jul 22 16:41:29.856673 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:29.856569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" event={"ID":"88c8d142-3b71-4623-be8a-d258bbfda6ca","Type":"ContainerDied","Data":"5e94225f666ef86558e97a2b0970be24ca159053d0e65661a98bdc01e38ad31d"} Jul 22 16:41:29.937431 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:29.937407 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:41:30.042092 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.042059 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c8d142-3b71-4623-be8a-d258bbfda6ca-extensions-socket-volume\") pod \"88c8d142-3b71-4623-be8a-d258bbfda6ca\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " Jul 22 16:41:30.042270 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.042123 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt7hb\" (UniqueName: \"kubernetes.io/projected/88c8d142-3b71-4623-be8a-d258bbfda6ca-kube-api-access-nt7hb\") pod \"88c8d142-3b71-4623-be8a-d258bbfda6ca\" (UID: \"88c8d142-3b71-4623-be8a-d258bbfda6ca\") " Jul 22 16:41:30.042459 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.042433 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c8d142-3b71-4623-be8a-d258bbfda6ca-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "88c8d142-3b71-4623-be8a-d258bbfda6ca" (UID: "88c8d142-3b71-4623-be8a-d258bbfda6ca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:41:30.042676 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.042652 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c8d142-3b71-4623-be8a-d258bbfda6ca-extensions-socket-volume\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:41:30.044263 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.044238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c8d142-3b71-4623-be8a-d258bbfda6ca-kube-api-access-nt7hb" (OuterVolumeSpecName: "kube-api-access-nt7hb") pod "88c8d142-3b71-4623-be8a-d258bbfda6ca" (UID: "88c8d142-3b71-4623-be8a-d258bbfda6ca"). InnerVolumeSpecName "kube-api-access-nt7hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:41:30.143330 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.143297 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt7hb\" (UniqueName: \"kubernetes.io/projected/88c8d142-3b71-4623-be8a-d258bbfda6ca-kube-api-access-nt7hb\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:41:30.864204 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.864179 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" Jul 22 16:41:30.864204 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.864185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm" event={"ID":"88c8d142-3b71-4623-be8a-d258bbfda6ca","Type":"ContainerDied","Data":"62ed89c0b6780093f8b8d039fdfaa5f46adf6c1c1967cf21965094c18ad86ec2"} Jul 22 16:41:30.864741 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.864240 2580 scope.go:117] "RemoveContainer" containerID="5e94225f666ef86558e97a2b0970be24ca159053d0e65661a98bdc01e38ad31d" Jul 22 16:41:30.891904 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.891875 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm"] Jul 22 16:41:30.896232 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.896207 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-67kwm"] Jul 22 16:41:30.980133 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:41:30.980106 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c8d142-3b71-4623-be8a-d258bbfda6ca" path="/var/lib/kubelet/pods/88c8d142-3b71-4623-be8a-d258bbfda6ca/volumes" Jul 22 16:42:29.269667 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.269632 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz"] Jul 22 16:42:29.270186 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270119 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c8d142-3b71-4623-be8a-d258bbfda6ca" containerName="manager" Jul 22 16:42:29.270186 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270131 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c8d142-3b71-4623-be8a-d258bbfda6ca" containerName="manager" Jul 22 16:42:29.270186 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270143 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548ff583-c4b3-4623-8d40-eb176437231a" containerName="authorino" Jul 22 16:42:29.270186 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270148 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="548ff583-c4b3-4623-8d40-eb176437231a" containerName="authorino" Jul 22 16:42:29.270186 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270155 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5f43d9-2743-4849-8100-2e73fcd2563b" containerName="manager" Jul 22 16:42:29.270186 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270160 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5f43d9-2743-4849-8100-2e73fcd2563b" containerName="manager" Jul 22 16:42:29.270381 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c8d142-3b71-4623-be8a-d258bbfda6ca" containerName="manager" Jul 22 16:42:29.270381 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270234 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="548ff583-c4b3-4623-8d40-eb176437231a" containerName="authorino" Jul 22 16:42:29.270381 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.270241 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5f43d9-2743-4849-8100-2e73fcd2563b" containerName="manager" Jul 22 16:42:29.273531 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.273514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.276071 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.276049 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mqv9r\"" Jul 22 16:42:29.285389 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.285359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz"] Jul 22 16:42:29.376539 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.376501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ed05e73-2d51-4dd3-990e-30a7d01694ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g8nvz\" (UID: \"8ed05e73-2d51-4dd3-990e-30a7d01694ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.376539 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.376547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphr5\" (UniqueName: \"kubernetes.io/projected/8ed05e73-2d51-4dd3-990e-30a7d01694ad-kube-api-access-jphr5\") pod \"kuadrant-operator-controller-manager-86878c769f-g8nvz\" (UID: \"8ed05e73-2d51-4dd3-990e-30a7d01694ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.477938 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.477901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ed05e73-2d51-4dd3-990e-30a7d01694ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g8nvz\" (UID: \"8ed05e73-2d51-4dd3-990e-30a7d01694ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.478138 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.477959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphr5\" (UniqueName: \"kubernetes.io/projected/8ed05e73-2d51-4dd3-990e-30a7d01694ad-kube-api-access-jphr5\") pod \"kuadrant-operator-controller-manager-86878c769f-g8nvz\" (UID: \"8ed05e73-2d51-4dd3-990e-30a7d01694ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.478389 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.478357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ed05e73-2d51-4dd3-990e-30a7d01694ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g8nvz\" (UID: \"8ed05e73-2d51-4dd3-990e-30a7d01694ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.488875 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.488846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphr5\" (UniqueName: \"kubernetes.io/projected/8ed05e73-2d51-4dd3-990e-30a7d01694ad-kube-api-access-jphr5\") pod \"kuadrant-operator-controller-manager-86878c769f-g8nvz\" (UID: \"8ed05e73-2d51-4dd3-990e-30a7d01694ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.584626 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.584517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:29.726467 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:29.726441 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz"] Jul 22 16:42:29.728127 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:42:29.728097 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed05e73_2d51_4dd3_990e_30a7d01694ad.slice/crio-2dea2b7de1a3ad70fabda39fcf89aae138fb8f6023bdd59f061a30db151eb00f WatchSource:0}: Error finding container 2dea2b7de1a3ad70fabda39fcf89aae138fb8f6023bdd59f061a30db151eb00f: Status 404 returned error can't find the container with id 2dea2b7de1a3ad70fabda39fcf89aae138fb8f6023bdd59f061a30db151eb00f Jul 22 16:42:30.207065 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:30.207027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" event={"ID":"8ed05e73-2d51-4dd3-990e-30a7d01694ad","Type":"ContainerStarted","Data":"eff0689fd2d0f4b41994bf724898cc26df0bd8d81dbc72cee33337a7bd37aa48"} Jul 22 16:42:30.207261 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:30.207082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" event={"ID":"8ed05e73-2d51-4dd3-990e-30a7d01694ad","Type":"ContainerStarted","Data":"2dea2b7de1a3ad70fabda39fcf89aae138fb8f6023bdd59f061a30db151eb00f"} Jul 22 16:42:30.207261 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:30.207107 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:42:30.230849 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:30.230794 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" podStartSLOduration=1.230777721 podStartE2EDuration="1.230777721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:42:30.229776024 +0000 UTC m=+1307.809869946" watchObservedRunningTime="2026-07-22 16:42:30.230777721 +0000 UTC m=+1307.810871615" Jul 22 16:42:41.216031 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:42:41.216000 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8nvz" Jul 22 16:50:25.689310 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.689263 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm"] Jul 22 16:50:25.693759 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.693738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.695939 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.695917 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c9720566-serving-cert\"" Jul 22 16:50:25.703548 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.703515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm"] Jul 22 16:50:25.761467 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.761419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bf6f7-ab6a-4773-b029-ae3a6b533e42-maas-api-tls\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.761467 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.761471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bf6f7-ab6a-4773-b029-ae3a6b533e42-openshift-service-ca\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.761697 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.761558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlrt\" (UniqueName: \"kubernetes.io/projected/818bf6f7-ab6a-4773-b029-ae3a6b533e42-kube-api-access-9xlrt\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.862356 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.862316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bf6f7-ab6a-4773-b029-ae3a6b533e42-maas-api-tls\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.862563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.862460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bf6f7-ab6a-4773-b029-ae3a6b533e42-openshift-service-ca\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.862616 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.862562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlrt\" (UniqueName: \"kubernetes.io/projected/818bf6f7-ab6a-4773-b029-ae3a6b533e42-kube-api-access-9xlrt\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.863101 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.863076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bf6f7-ab6a-4773-b029-ae3a6b533e42-openshift-service-ca\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.864861 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.864840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bf6f7-ab6a-4773-b029-ae3a6b533e42-maas-api-tls\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:25.872076 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:25.872050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlrt\" (UniqueName: \"kubernetes.io/projected/818bf6f7-ab6a-4773-b029-ae3a6b533e42-kube-api-access-9xlrt\") pod \"maas-api-e2e-ait-c9720566-dbc449745-r4xbm\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:26.006563 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:26.006531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:26.151109 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:26.151040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm"] Jul 22 16:50:26.154083 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:50:26.154045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818bf6f7_ab6a_4773_b029_ae3a6b533e42.slice/crio-9f01a135695b47cfcf072ad9aaa9cb9930f0867fb3ddfbc1368c2bcb4bc0b4c5 WatchSource:0}: Error finding container 9f01a135695b47cfcf072ad9aaa9cb9930f0867fb3ddfbc1368c2bcb4bc0b4c5: Status 404 returned error can't find the container with id 9f01a135695b47cfcf072ad9aaa9cb9930f0867fb3ddfbc1368c2bcb4bc0b4c5 Jul 22 16:50:26.155803 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:26.155786 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:50:27.029612 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:27.029579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" event={"ID":"818bf6f7-ab6a-4773-b029-ae3a6b533e42","Type":"ContainerStarted","Data":"4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100"} Jul 22 16:50:27.030034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:27.029628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" event={"ID":"818bf6f7-ab6a-4773-b029-ae3a6b533e42","Type":"ContainerStarted","Data":"9f01a135695b47cfcf072ad9aaa9cb9930f0867fb3ddfbc1368c2bcb4bc0b4c5"} Jul 22 16:50:27.030034 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:27.029718 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:27.048855 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:27.048811 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" podStartSLOduration=1.592289863 podStartE2EDuration="2.048797057s" totalImagesPullingTime="456.507194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:50:27.047458337 +0000 UTC m=+1784.627552255" watchObservedRunningTime="2026-07-22 16:50:27.048797057 +0000 UTC m=+1784.628890987" Jul 22 16:50:33.042753 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:33.042717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:50:51.272781 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.272747 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx"] Jul 22 16:50:51.276472 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.276452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:51.278674 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.278654 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xmwmv\"" Jul 22 16:50:51.286349 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.286327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx"] Jul 22 16:50:51.410588 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.410555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ff40e2-e0f1-423f-a5ff-6f0184edf395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9720566-t8lvx\" (UID: \"d2ff40e2-e0f1-423f-a5ff-6f0184edf395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:51.511830 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.511798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ff40e2-e0f1-423f-a5ff-6f0184edf395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9720566-t8lvx\" (UID: \"d2ff40e2-e0f1-423f-a5ff-6f0184edf395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:51.512449 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.512422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ff40e2-e0f1-423f-a5ff-6f0184edf395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9720566-t8lvx\" (UID: \"d2ff40e2-e0f1-423f-a5ff-6f0184edf395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:51.588319 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.588235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:51.746616 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:51.746586 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx"] Jul 22 16:50:51.748881 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:50:51.748853 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ff40e2_e0f1_423f_a5ff_6f0184edf395.slice/crio-35a87509da85086d8cdc5cb49c49692bea8623f9fc5f1c30a3536f63815541ff WatchSource:0}: Error finding container 35a87509da85086d8cdc5cb49c49692bea8623f9fc5f1c30a3536f63815541ff: Status 404 returned error can't find the container with id 35a87509da85086d8cdc5cb49c49692bea8623f9fc5f1c30a3536f63815541ff Jul 22 16:50:52.181707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:52.181672 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" event={"ID":"d2ff40e2-e0f1-423f-a5ff-6f0184edf395","Type":"ContainerStarted","Data":"35a87509da85086d8cdc5cb49c49692bea8623f9fc5f1c30a3536f63815541ff"} Jul 22 16:50:55.202123 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:55.202087 2580 generic.go:356] "Generic (PLEG): container finished" podID="d2ff40e2-e0f1-423f-a5ff-6f0184edf395" containerID="ad9b5d91b24f67435d70aff5ea54c3f8eaa5403c8e1605989a1d7ed665da938b" exitCode=0 Jul 22 16:50:55.202642 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:55.202138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" event={"ID":"d2ff40e2-e0f1-423f-a5ff-6f0184edf395","Type":"ContainerDied","Data":"ad9b5d91b24f67435d70aff5ea54c3f8eaa5403c8e1605989a1d7ed665da938b"} Jul 22 16:50:56.346000 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:56.345975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:56.458150 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:56.458064 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ff40e2-e0f1-423f-a5ff-6f0184edf395-maas-api-service-ca\") pod \"d2ff40e2-e0f1-423f-a5ff-6f0184edf395\" (UID: \"d2ff40e2-e0f1-423f-a5ff-6f0184edf395\") " Jul 22 16:50:56.458451 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:56.458428 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ff40e2-e0f1-423f-a5ff-6f0184edf395-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2ff40e2-e0f1-423f-a5ff-6f0184edf395" (UID: "d2ff40e2-e0f1-423f-a5ff-6f0184edf395"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:50:56.559004 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:56.558967 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ff40e2-e0f1-423f-a5ff-6f0184edf395-maas-api-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:50:57.213989 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:57.213961 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" Jul 22 16:50:57.214160 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:57.213963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx" event={"ID":"d2ff40e2-e0f1-423f-a5ff-6f0184edf395","Type":"ContainerDied","Data":"35a87509da85086d8cdc5cb49c49692bea8623f9fc5f1c30a3536f63815541ff"} Jul 22 16:50:57.214160 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:50:57.214089 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35a87509da85086d8cdc5cb49c49692bea8623f9fc5f1c30a3536f63815541ff" Jul 22 16:51:01.840433 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:01.840383 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm"] Jul 22 16:51:01.840854 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:01.840616 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" podUID="818bf6f7-ab6a-4773-b029-ae3a6b533e42" containerName="maas-api" containerID="cri-o://4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100" gracePeriod=30 Jul 22 16:51:02.097796 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.097731 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:51:02.213040 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.213005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bf6f7-ab6a-4773-b029-ae3a6b533e42-maas-api-tls\") pod \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " Jul 22 16:51:02.213215 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.213085 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xlrt\" (UniqueName: \"kubernetes.io/projected/818bf6f7-ab6a-4773-b029-ae3a6b533e42-kube-api-access-9xlrt\") pod \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " Jul 22 16:51:02.213215 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.213123 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bf6f7-ab6a-4773-b029-ae3a6b533e42-openshift-service-ca\") pod \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\" (UID: \"818bf6f7-ab6a-4773-b029-ae3a6b533e42\") " Jul 22 16:51:02.213532 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.213507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818bf6f7-ab6a-4773-b029-ae3a6b533e42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "818bf6f7-ab6a-4773-b029-ae3a6b533e42" (UID: "818bf6f7-ab6a-4773-b029-ae3a6b533e42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:51:02.215360 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.215330 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818bf6f7-ab6a-4773-b029-ae3a6b533e42-kube-api-access-9xlrt" (OuterVolumeSpecName: "kube-api-access-9xlrt") pod "818bf6f7-ab6a-4773-b029-ae3a6b533e42" (UID: "818bf6f7-ab6a-4773-b029-ae3a6b533e42"). InnerVolumeSpecName "kube-api-access-9xlrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:51:02.215516 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.215360 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818bf6f7-ab6a-4773-b029-ae3a6b533e42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "818bf6f7-ab6a-4773-b029-ae3a6b533e42" (UID: "818bf6f7-ab6a-4773-b029-ae3a6b533e42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:51:02.244537 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.244508 2580 generic.go:356] "Generic (PLEG): container finished" podID="818bf6f7-ab6a-4773-b029-ae3a6b533e42" containerID="4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100" exitCode=0 Jul 22 16:51:02.244665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.244553 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" Jul 22 16:51:02.244665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.244597 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" event={"ID":"818bf6f7-ab6a-4773-b029-ae3a6b533e42","Type":"ContainerDied","Data":"4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100"} Jul 22 16:51:02.244665 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.244656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm" event={"ID":"818bf6f7-ab6a-4773-b029-ae3a6b533e42","Type":"ContainerDied","Data":"9f01a135695b47cfcf072ad9aaa9cb9930f0867fb3ddfbc1368c2bcb4bc0b4c5"} Jul 22 16:51:02.244831 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.244685 2580 scope.go:117] "RemoveContainer" containerID="4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100" Jul 22 16:51:02.255588 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.255571 2580 scope.go:117] "RemoveContainer" containerID="4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100" Jul 22 16:51:02.255830 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:51:02.255812 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100\": container with ID starting with 4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100 not found: ID does not exist" containerID="4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100" Jul 22 16:51:02.255886 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.255839 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100"} err="failed to get container status \"4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100\": rpc error: code = NotFound desc = could not find container \"4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100\": container with ID starting with 4c03e342ea2c7c2cd1476abbdcfb1402e1b659d94dc11fa75555035d6bdb6100 not found: ID does not exist" Jul 22 16:51:02.271333 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.271293 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm"] Jul 22 16:51:02.278612 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.278590 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9720566-dbc449745-r4xbm"] Jul 22 16:51:02.314160 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.314127 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlrt\" (UniqueName: \"kubernetes.io/projected/818bf6f7-ab6a-4773-b029-ae3a6b533e42-kube-api-access-9xlrt\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:51:02.314289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.314160 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bf6f7-ab6a-4773-b029-ae3a6b533e42-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:51:02.314289 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.314178 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bf6f7-ab6a-4773-b029-ae3a6b533e42-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:51:02.980723 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:02.980687 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="818bf6f7-ab6a-4773-b029-ae3a6b533e42" path="/var/lib/kubelet/pods/818bf6f7-ab6a-4773-b029-ae3a6b533e42/volumes" Jul 22 16:51:06.370181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:06.370137 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx"] Jul 22 16:51:06.373114 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:06.373093 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9720566-t8lvx"] Jul 22 16:51:06.980578 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:06.980546 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ff40e2-e0f1-423f-a5ff-6f0184edf395" path="/var/lib/kubelet/pods/d2ff40e2-e0f1-423f-a5ff-6f0184edf395/volumes" Jul 22 16:51:10.105644 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.105610 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph"] Jul 22 16:51:10.106105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.106090 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ff40e2-e0f1-423f-a5ff-6f0184edf395" containerName="revoke-keys" Jul 22 16:51:10.106105 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.106105 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ff40e2-e0f1-423f-a5ff-6f0184edf395" containerName="revoke-keys" Jul 22 16:51:10.106233 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.106116 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818bf6f7-ab6a-4773-b029-ae3a6b533e42" containerName="maas-api" Jul 22 16:51:10.106233 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.106122 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="818bf6f7-ab6a-4773-b029-ae3a6b533e42" containerName="maas-api" Jul 22 16:51:10.106233 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.106201 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="818bf6f7-ab6a-4773-b029-ae3a6b533e42" containerName="maas-api" Jul 22 16:51:10.106233 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.106215 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ff40e2-e0f1-423f-a5ff-6f0184edf395" containerName="revoke-keys" Jul 22 16:51:10.110758 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.110728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.113508 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.113282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-30e2948f-serving-cert\"" Jul 22 16:51:10.119452 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.119425 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph"] Jul 22 16:51:10.183578 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.181081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60699e89-ed17-4181-ab4f-8096dab4df40-maas-api-tls\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.183578 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.181209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60699e89-ed17-4181-ab4f-8096dab4df40-openshift-service-ca\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.183578 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.181263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4t4\" (UniqueName: \"kubernetes.io/projected/60699e89-ed17-4181-ab4f-8096dab4df40-kube-api-access-nd4t4\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.282661 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.282626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60699e89-ed17-4181-ab4f-8096dab4df40-openshift-service-ca\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.282661 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.282664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4t4\" (UniqueName: \"kubernetes.io/projected/60699e89-ed17-4181-ab4f-8096dab4df40-kube-api-access-nd4t4\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.282877 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.282737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60699e89-ed17-4181-ab4f-8096dab4df40-maas-api-tls\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.283269 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.283248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60699e89-ed17-4181-ab4f-8096dab4df40-openshift-service-ca\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.285152 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.285128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60699e89-ed17-4181-ab4f-8096dab4df40-maas-api-tls\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.291018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.290996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4t4\" (UniqueName: \"kubernetes.io/projected/60699e89-ed17-4181-ab4f-8096dab4df40-kube-api-access-nd4t4\") pod \"maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.425643 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.425561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:10.578601 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:10.578572 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph"] Jul 22 16:51:10.579971 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:51:10.579942 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60699e89_ed17_4181_ab4f_8096dab4df40.slice/crio-5c9583832f6c3e15ed4dbc73686bebf12299de878db0105ade8d61558e19f868 WatchSource:0}: Error finding container 5c9583832f6c3e15ed4dbc73686bebf12299de878db0105ade8d61558e19f868: Status 404 returned error can't find the container with id 5c9583832f6c3e15ed4dbc73686bebf12299de878db0105ade8d61558e19f868 Jul 22 16:51:11.301320 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:11.301285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" event={"ID":"60699e89-ed17-4181-ab4f-8096dab4df40","Type":"ContainerStarted","Data":"9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48"} Jul 22 16:51:11.301737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:11.301339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" event={"ID":"60699e89-ed17-4181-ab4f-8096dab4df40","Type":"ContainerStarted","Data":"5c9583832f6c3e15ed4dbc73686bebf12299de878db0105ade8d61558e19f868"} Jul 22 16:51:11.301737 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:11.301423 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:11.321559 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:11.321511 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" podStartSLOduration=1.026233584 podStartE2EDuration="1.321495918s" totalImagesPullingTime="295.262334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:51:11.318245152 +0000 UTC m=+1828.898339067" watchObservedRunningTime="2026-07-22 16:51:11.321495918 +0000 UTC m=+1828.901589814" Jul 22 16:51:17.312842 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:17.312810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:45.469000 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.468960 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph"] Jul 22 16:51:45.469549 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.469224 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" podUID="60699e89-ed17-4181-ab4f-8096dab4df40" containerName="maas-api" containerID="cri-o://9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48" gracePeriod=30 Jul 22 16:51:45.724032 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.723967 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:45.834959 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.834918 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60699e89-ed17-4181-ab4f-8096dab4df40-maas-api-tls\") pod \"60699e89-ed17-4181-ab4f-8096dab4df40\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " Jul 22 16:51:45.835148 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.835023 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd4t4\" (UniqueName: \"kubernetes.io/projected/60699e89-ed17-4181-ab4f-8096dab4df40-kube-api-access-nd4t4\") pod \"60699e89-ed17-4181-ab4f-8096dab4df40\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " Jul 22 16:51:45.835148 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.835063 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60699e89-ed17-4181-ab4f-8096dab4df40-openshift-service-ca\") pod \"60699e89-ed17-4181-ab4f-8096dab4df40\" (UID: \"60699e89-ed17-4181-ab4f-8096dab4df40\") " Jul 22 16:51:45.835678 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.835651 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60699e89-ed17-4181-ab4f-8096dab4df40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60699e89-ed17-4181-ab4f-8096dab4df40" (UID: "60699e89-ed17-4181-ab4f-8096dab4df40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:51:45.837038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.837007 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60699e89-ed17-4181-ab4f-8096dab4df40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60699e89-ed17-4181-ab4f-8096dab4df40" (UID: "60699e89-ed17-4181-ab4f-8096dab4df40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:51:45.837487 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.837468 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60699e89-ed17-4181-ab4f-8096dab4df40-kube-api-access-nd4t4" (OuterVolumeSpecName: "kube-api-access-nd4t4") pod "60699e89-ed17-4181-ab4f-8096dab4df40" (UID: "60699e89-ed17-4181-ab4f-8096dab4df40"). InnerVolumeSpecName "kube-api-access-nd4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:51:45.936204 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.936166 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60699e89-ed17-4181-ab4f-8096dab4df40-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:51:45.936204 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.936199 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4t4\" (UniqueName: \"kubernetes.io/projected/60699e89-ed17-4181-ab4f-8096dab4df40-kube-api-access-nd4t4\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:51:45.936204 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:45.936209 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60699e89-ed17-4181-ab4f-8096dab4df40-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:51:46.504623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.504585 2580 generic.go:356] "Generic (PLEG): container finished" podID="60699e89-ed17-4181-ab4f-8096dab4df40" containerID="9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48" exitCode=0 Jul 22 16:51:46.505042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.504627 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" Jul 22 16:51:46.505042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.504659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" event={"ID":"60699e89-ed17-4181-ab4f-8096dab4df40","Type":"ContainerDied","Data":"9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48"} Jul 22 16:51:46.505042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.504704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph" event={"ID":"60699e89-ed17-4181-ab4f-8096dab4df40","Type":"ContainerDied","Data":"5c9583832f6c3e15ed4dbc73686bebf12299de878db0105ade8d61558e19f868"} Jul 22 16:51:46.505042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.504728 2580 scope.go:117] "RemoveContainer" containerID="9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48" Jul 22 16:51:46.514641 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.514624 2580 scope.go:117] "RemoveContainer" containerID="9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48" Jul 22 16:51:46.514898 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:51:46.514880 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48\": container with ID starting with 9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48 not found: ID does not exist" containerID="9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48" Jul 22 16:51:46.514958 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.514907 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48"} err="failed to get container status \"9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48\": rpc error: code = NotFound desc = could not find container \"9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48\": container with ID starting with 9b2f0ea4ca75313e5d831b942d3a1d90fd4d0233455deedf0d432c7b22142d48 not found: ID does not exist" Jul 22 16:51:46.526775 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.526740 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph"] Jul 22 16:51:46.529630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.529609 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-30e2948f-76586656f7-vl2ph"] Jul 22 16:51:46.981165 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:46.981079 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60699e89-ed17-4181-ab4f-8096dab4df40" path="/var/lib/kubelet/pods/60699e89-ed17-4181-ab4f-8096dab4df40/volumes" Jul 22 16:51:54.135283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.135249 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn"] Jul 22 16:51:54.136683 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.136649 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60699e89-ed17-4181-ab4f-8096dab4df40" containerName="maas-api" Jul 22 16:51:54.136683 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.136678 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60699e89-ed17-4181-ab4f-8096dab4df40" containerName="maas-api" Jul 22 16:51:54.136881 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.136793 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60699e89-ed17-4181-ab4f-8096dab4df40" containerName="maas-api" Jul 22 16:51:54.141686 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.141667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.143954 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.143929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-66fe2bd0-serving-cert\"" Jul 22 16:51:54.149676 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.149650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn"] Jul 22 16:51:54.213501 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.213475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/026a268a-a6b9-4079-bc72-01d9ea6fc882-openshift-service-ca\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.213652 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.213521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.213652 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.213618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvhd\" (UniqueName: \"kubernetes.io/projected/026a268a-a6b9-4079-bc72-01d9ea6fc882-kube-api-access-mnvhd\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.314759 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.314729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvhd\" (UniqueName: \"kubernetes.io/projected/026a268a-a6b9-4079-bc72-01d9ea6fc882-kube-api-access-mnvhd\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.314940 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.314818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/026a268a-a6b9-4079-bc72-01d9ea6fc882-openshift-service-ca\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.314940 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.314853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.315086 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:51:54.314951 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-serving-cert: secret "maas-api-e2e-ait-66fe2bd0-serving-cert" not found Jul 22 16:51:54.315086 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:51:54.315013 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls podName:026a268a-a6b9-4079-bc72-01d9ea6fc882 nodeName:}" failed. No retries permitted until 2026-07-22 16:51:54.814995196 +0000 UTC m=+1872.395089073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls") pod "maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" (UID: "026a268a-a6b9-4079-bc72-01d9ea6fc882") : secret "maas-api-e2e-ait-66fe2bd0-serving-cert" not found Jul 22 16:51:54.315576 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.315557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/026a268a-a6b9-4079-bc72-01d9ea6fc882-openshift-service-ca\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.325557 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.325529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvhd\" (UniqueName: \"kubernetes.io/projected/026a268a-a6b9-4079-bc72-01d9ea6fc882-kube-api-access-mnvhd\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.819921 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.819885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:54.822405 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:54.822369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls\") pod \"maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:55.053430 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:55.053383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:55.197369 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:55.197341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn"] Jul 22 16:51:55.198544 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:51:55.198517 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026a268a_a6b9_4079_bc72_01d9ea6fc882.slice/crio-c06295388634839f130ac4e01cfeb5ee3d140e7d6752bcf9f1b1f60253d9218d WatchSource:0}: Error finding container c06295388634839f130ac4e01cfeb5ee3d140e7d6752bcf9f1b1f60253d9218d: Status 404 returned error can't find the container with id c06295388634839f130ac4e01cfeb5ee3d140e7d6752bcf9f1b1f60253d9218d Jul 22 16:51:55.561227 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:55.561184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" event={"ID":"026a268a-a6b9-4079-bc72-01d9ea6fc882","Type":"ContainerStarted","Data":"c06295388634839f130ac4e01cfeb5ee3d140e7d6752bcf9f1b1f60253d9218d"} Jul 22 16:51:56.568582 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:56.568542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" event={"ID":"026a268a-a6b9-4079-bc72-01d9ea6fc882","Type":"ContainerStarted","Data":"c225c51bb72774ba70980c6558ce3ee2dec3f9b937687d1447ae03a81b37b35d"} Jul 22 16:51:56.568998 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:56.568621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:51:56.588027 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:51:56.587983 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" podStartSLOduration=2.208467915 podStartE2EDuration="2.58796848s" totalImagesPullingTime="379.500565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:51:56.585128228 +0000 UTC m=+1874.165222126" watchObservedRunningTime="2026-07-22 16:51:56.58796848 +0000 UTC m=+1874.168062377" Jul 22 16:52:02.580427 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:02.580371 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:52:30.552319 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.552277 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn"] Jul 22 16:52:30.555012 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.552722 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" podUID="026a268a-a6b9-4079-bc72-01d9ea6fc882" containerName="maas-api" containerID="cri-o://c225c51bb72774ba70980c6558ce3ee2dec3f9b937687d1447ae03a81b37b35d" gracePeriod=30 Jul 22 16:52:30.781092 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.781057 2580 generic.go:356] "Generic (PLEG): container finished" podID="026a268a-a6b9-4079-bc72-01d9ea6fc882" containerID="c225c51bb72774ba70980c6558ce3ee2dec3f9b937687d1447ae03a81b37b35d" exitCode=0 Jul 22 16:52:30.781242 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.781125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" event={"ID":"026a268a-a6b9-4079-bc72-01d9ea6fc882","Type":"ContainerDied","Data":"c225c51bb72774ba70980c6558ce3ee2dec3f9b937687d1447ae03a81b37b35d"} Jul 22 16:52:30.807038 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.806979 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:52:30.865162 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.865127 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/026a268a-a6b9-4079-bc72-01d9ea6fc882-openshift-service-ca\") pod \"026a268a-a6b9-4079-bc72-01d9ea6fc882\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " Jul 22 16:52:30.865327 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.865191 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls\") pod \"026a268a-a6b9-4079-bc72-01d9ea6fc882\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " Jul 22 16:52:30.865327 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.865234 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnvhd\" (UniqueName: \"kubernetes.io/projected/026a268a-a6b9-4079-bc72-01d9ea6fc882-kube-api-access-mnvhd\") pod \"026a268a-a6b9-4079-bc72-01d9ea6fc882\" (UID: \"026a268a-a6b9-4079-bc72-01d9ea6fc882\") " Jul 22 16:52:30.865619 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.865590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026a268a-a6b9-4079-bc72-01d9ea6fc882-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "026a268a-a6b9-4079-bc72-01d9ea6fc882" (UID: "026a268a-a6b9-4079-bc72-01d9ea6fc882"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:52:30.867304 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.867279 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "026a268a-a6b9-4079-bc72-01d9ea6fc882" (UID: "026a268a-a6b9-4079-bc72-01d9ea6fc882"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:52:30.867424 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.867322 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a268a-a6b9-4079-bc72-01d9ea6fc882-kube-api-access-mnvhd" (OuterVolumeSpecName: "kube-api-access-mnvhd") pod "026a268a-a6b9-4079-bc72-01d9ea6fc882" (UID: "026a268a-a6b9-4079-bc72-01d9ea6fc882"). InnerVolumeSpecName "kube-api-access-mnvhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:52:30.966841 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.966796 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/026a268a-a6b9-4079-bc72-01d9ea6fc882-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:52:30.966841 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.966832 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnvhd\" (UniqueName: \"kubernetes.io/projected/026a268a-a6b9-4079-bc72-01d9ea6fc882-kube-api-access-mnvhd\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:52:30.966841 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:30.966844 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/026a268a-a6b9-4079-bc72-01d9ea6fc882-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:52:31.788636 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:31.788606 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" Jul 22 16:52:31.789054 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:31.788632 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn" event={"ID":"026a268a-a6b9-4079-bc72-01d9ea6fc882","Type":"ContainerDied","Data":"c06295388634839f130ac4e01cfeb5ee3d140e7d6752bcf9f1b1f60253d9218d"} Jul 22 16:52:31.789054 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:31.788694 2580 scope.go:117] "RemoveContainer" containerID="c225c51bb72774ba70980c6558ce3ee2dec3f9b937687d1447ae03a81b37b35d" Jul 22 16:52:31.810850 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:31.810827 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn"] Jul 22 16:52:31.817232 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:31.817204 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66fe2bd0-7d6fdf5d4b-mfqdn"] Jul 22 16:52:32.979621 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:32.979589 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="026a268a-a6b9-4079-bc72-01d9ea6fc882" path="/var/lib/kubelet/pods/026a268a-a6b9-4079-bc72-01d9ea6fc882/volumes" Jul 22 16:52:40.346462 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.346423 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs"] Jul 22 16:52:40.347096 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.347076 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026a268a-a6b9-4079-bc72-01d9ea6fc882" containerName="maas-api" Jul 22 16:52:40.347096 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.347098 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="026a268a-a6b9-4079-bc72-01d9ea6fc882" containerName="maas-api" Jul 22 16:52:40.347236 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.347227 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="026a268a-a6b9-4079-bc72-01d9ea6fc882" containerName="maas-api" Jul 22 16:52:40.352013 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.351988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.354283 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.354262 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d9102441-serving-cert\"" Jul 22 16:52:40.365526 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.365499 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs"] Jul 22 16:52:40.462801 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.462770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.462975 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.462917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5d91cb-f7ce-4209-9057-c77110345a8a-openshift-service-ca\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.463071 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.462976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d5l6\" (UniqueName: \"kubernetes.io/projected/ed5d91cb-f7ce-4209-9057-c77110345a8a-kube-api-access-9d5l6\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.564471 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.564431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d5l6\" (UniqueName: \"kubernetes.io/projected/ed5d91cb-f7ce-4209-9057-c77110345a8a-kube-api-access-9d5l6\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.564673 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.564498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.564673 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.564574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5d91cb-f7ce-4209-9057-c77110345a8a-openshift-service-ca\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.564673 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:52:40.564655 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-serving-cert: secret "maas-api-e2e-derive-d9102441-serving-cert" not found Jul 22 16:52:40.564844 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:52:40.564715 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls podName:ed5d91cb-f7ce-4209-9057-c77110345a8a nodeName:}" failed. No retries permitted until 2026-07-22 16:52:41.06469683 +0000 UTC m=+1918.644790709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls") pod "maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" (UID: "ed5d91cb-f7ce-4209-9057-c77110345a8a") : secret "maas-api-e2e-derive-d9102441-serving-cert" not found Jul 22 16:52:40.565149 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.565132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5d91cb-f7ce-4209-9057-c77110345a8a-openshift-service-ca\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:40.573005 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:40.572977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d5l6\" (UniqueName: \"kubernetes.io/projected/ed5d91cb-f7ce-4209-9057-c77110345a8a-kube-api-access-9d5l6\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:41.069815 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:41.069774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:41.072323 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:41.072290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls\") pod \"maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:41.263247 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:41.263210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:41.405959 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:41.405926 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs"] Jul 22 16:52:41.407478 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:52:41.407444 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5d91cb_f7ce_4209_9057_c77110345a8a.slice/crio-e0e4b70fef9bb5b970f7cb8ca46b46befa5b43ee3fc828baad7cb192c0e435c8 WatchSource:0}: Error finding container e0e4b70fef9bb5b970f7cb8ca46b46befa5b43ee3fc828baad7cb192c0e435c8: Status 404 returned error can't find the container with id e0e4b70fef9bb5b970f7cb8ca46b46befa5b43ee3fc828baad7cb192c0e435c8 Jul 22 16:52:41.852460 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:41.852428 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" event={"ID":"ed5d91cb-f7ce-4209-9057-c77110345a8a","Type":"ContainerStarted","Data":"e0e4b70fef9bb5b970f7cb8ca46b46befa5b43ee3fc828baad7cb192c0e435c8"} Jul 22 16:52:42.862348 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:42.862302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" event={"ID":"ed5d91cb-f7ce-4209-9057-c77110345a8a","Type":"ContainerStarted","Data":"c5e0c24617578d6a6c42c917160991f0be6a18cf83b510021c10a3585dac582d"} Jul 22 16:52:42.862860 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:42.862534 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:52:42.880839 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:42.880797 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" podStartSLOduration=2.467321087 podStartE2EDuration="2.880783835s" totalImagesPullingTime="413.462748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:52:42.879033867 +0000 UTC m=+1920.459127782" watchObservedRunningTime="2026-07-22 16:52:42.880783835 +0000 UTC m=+1920.460877784" Jul 22 16:52:48.874157 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:52:48.874126 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:53:05.259781 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.259747 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x"] Jul 22 16:53:05.263484 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.263467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:05.265628 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.265602 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xmwmv\"" Jul 22 16:53:05.271782 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.271756 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x"] Jul 22 16:53:05.304439 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.304375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb87d48-bf9f-4fca-959d-fb23ebfc107c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d9102441-2mx4x\" (UID: \"1fb87d48-bf9f-4fca-959d-fb23ebfc107c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:05.406055 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.406023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb87d48-bf9f-4fca-959d-fb23ebfc107c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d9102441-2mx4x\" (UID: \"1fb87d48-bf9f-4fca-959d-fb23ebfc107c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:05.406626 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.406609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb87d48-bf9f-4fca-959d-fb23ebfc107c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d9102441-2mx4x\" (UID: \"1fb87d48-bf9f-4fca-959d-fb23ebfc107c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:05.576389 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.576312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:05.714453 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:05.714428 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x"] Jul 22 16:53:05.715956 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:53:05.715929 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb87d48_bf9f_4fca_959d_fb23ebfc107c.slice/crio-10f5ccd9395f2efbd39b2ae6ded15fb80953a9c989e45a1e148ffc56dfff3505 WatchSource:0}: Error finding container 10f5ccd9395f2efbd39b2ae6ded15fb80953a9c989e45a1e148ffc56dfff3505: Status 404 returned error can't find the container with id 10f5ccd9395f2efbd39b2ae6ded15fb80953a9c989e45a1e148ffc56dfff3505 Jul 22 16:53:06.001841 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:06.001810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" event={"ID":"1fb87d48-bf9f-4fca-959d-fb23ebfc107c","Type":"ContainerStarted","Data":"10f5ccd9395f2efbd39b2ae6ded15fb80953a9c989e45a1e148ffc56dfff3505"} Jul 22 16:53:07.008946 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:07.008914 2580 generic.go:356] "Generic (PLEG): container finished" podID="1fb87d48-bf9f-4fca-959d-fb23ebfc107c" containerID="717246040ae1557cf2edfdc5de5efa5004678b79145bdbfec718f29fb01c2241" exitCode=0 Jul 22 16:53:07.009357 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:07.009001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" event={"ID":"1fb87d48-bf9f-4fca-959d-fb23ebfc107c","Type":"ContainerDied","Data":"717246040ae1557cf2edfdc5de5efa5004678b79145bdbfec718f29fb01c2241"} Jul 22 16:53:08.163911 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:08.163881 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:08.230681 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:08.230641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb87d48-bf9f-4fca-959d-fb23ebfc107c-maas-api-service-ca\") pod \"1fb87d48-bf9f-4fca-959d-fb23ebfc107c\" (UID: \"1fb87d48-bf9f-4fca-959d-fb23ebfc107c\") " Jul 22 16:53:08.230998 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:08.230975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb87d48-bf9f-4fca-959d-fb23ebfc107c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fb87d48-bf9f-4fca-959d-fb23ebfc107c" (UID: "1fb87d48-bf9f-4fca-959d-fb23ebfc107c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:53:08.332373 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:08.332292 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb87d48-bf9f-4fca-959d-fb23ebfc107c-maas-api-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:53:09.022202 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:09.022177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" Jul 22 16:53:09.022352 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:09.022215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x" event={"ID":"1fb87d48-bf9f-4fca-959d-fb23ebfc107c","Type":"ContainerDied","Data":"10f5ccd9395f2efbd39b2ae6ded15fb80953a9c989e45a1e148ffc56dfff3505"} Jul 22 16:53:09.022352 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:09.022263 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f5ccd9395f2efbd39b2ae6ded15fb80953a9c989e45a1e148ffc56dfff3505" Jul 22 16:53:15.853655 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:15.853619 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs"] Jul 22 16:53:15.854329 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:15.853903 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" podUID="ed5d91cb-f7ce-4209-9057-c77110345a8a" containerName="maas-api" containerID="cri-o://c5e0c24617578d6a6c42c917160991f0be6a18cf83b510021c10a3585dac582d" gracePeriod=30 Jul 22 16:53:16.065266 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.065235 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed5d91cb-f7ce-4209-9057-c77110345a8a" containerID="c5e0c24617578d6a6c42c917160991f0be6a18cf83b510021c10a3585dac582d" exitCode=0 Jul 22 16:53:16.065433 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.065310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" event={"ID":"ed5d91cb-f7ce-4209-9057-c77110345a8a","Type":"ContainerDied","Data":"c5e0c24617578d6a6c42c917160991f0be6a18cf83b510021c10a3585dac582d"} Jul 22 16:53:16.105167 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.105106 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:53:16.205462 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.205424 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls\") pod \"ed5d91cb-f7ce-4209-9057-c77110345a8a\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " Jul 22 16:53:16.205656 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.205478 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d5l6\" (UniqueName: \"kubernetes.io/projected/ed5d91cb-f7ce-4209-9057-c77110345a8a-kube-api-access-9d5l6\") pod \"ed5d91cb-f7ce-4209-9057-c77110345a8a\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " Jul 22 16:53:16.205656 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.205514 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5d91cb-f7ce-4209-9057-c77110345a8a-openshift-service-ca\") pod \"ed5d91cb-f7ce-4209-9057-c77110345a8a\" (UID: \"ed5d91cb-f7ce-4209-9057-c77110345a8a\") " Jul 22 16:53:16.205935 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.205909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5d91cb-f7ce-4209-9057-c77110345a8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed5d91cb-f7ce-4209-9057-c77110345a8a" (UID: "ed5d91cb-f7ce-4209-9057-c77110345a8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:53:16.207519 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.207496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed5d91cb-f7ce-4209-9057-c77110345a8a" (UID: "ed5d91cb-f7ce-4209-9057-c77110345a8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:53:16.207647 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.207626 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5d91cb-f7ce-4209-9057-c77110345a8a-kube-api-access-9d5l6" (OuterVolumeSpecName: "kube-api-access-9d5l6") pod "ed5d91cb-f7ce-4209-9057-c77110345a8a" (UID: "ed5d91cb-f7ce-4209-9057-c77110345a8a"). InnerVolumeSpecName "kube-api-access-9d5l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:53:16.307175 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.307143 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed5d91cb-f7ce-4209-9057-c77110345a8a-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:53:16.307175 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.307169 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d5l6\" (UniqueName: \"kubernetes.io/projected/ed5d91cb-f7ce-4209-9057-c77110345a8a-kube-api-access-9d5l6\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:53:16.307175 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:16.307178 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5d91cb-f7ce-4209-9057-c77110345a8a-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:53:17.073653 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:17.073624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" event={"ID":"ed5d91cb-f7ce-4209-9057-c77110345a8a","Type":"ContainerDied","Data":"e0e4b70fef9bb5b970f7cb8ca46b46befa5b43ee3fc828baad7cb192c0e435c8"} Jul 22 16:53:17.074099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:17.073682 2580 scope.go:117] "RemoveContainer" containerID="c5e0c24617578d6a6c42c917160991f0be6a18cf83b510021c10a3585dac582d" Jul 22 16:53:17.074099 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:17.073632 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs" Jul 22 16:53:17.092336 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:17.092309 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs"] Jul 22 16:53:17.096496 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:17.096472 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d9102441-6554f84c6c-g6rcs"] Jul 22 16:53:18.980317 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:18.980281 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed5d91cb-f7ce-4209-9057-c77110345a8a" path="/var/lib/kubelet/pods/ed5d91cb-f7ce-4209-9057-c77110345a8a/volumes" Jul 22 16:53:20.359494 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:20.359456 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x"] Jul 22 16:53:20.363334 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:20.363305 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9102441-2mx4x"] Jul 22 16:53:20.980478 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:53:20.980435 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb87d48-bf9f-4fca-959d-fb23ebfc107c" path="/var/lib/kubelet/pods/1fb87d48-bf9f-4fca-959d-fb23ebfc107c/volumes" Jul 22 16:55:04.631145 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631110 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j"] Jul 22 16:55:04.631797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631591 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb87d48-bf9f-4fca-959d-fb23ebfc107c" containerName="revoke-keys" Jul 22 16:55:04.631797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631604 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb87d48-bf9f-4fca-959d-fb23ebfc107c" containerName="revoke-keys" Jul 22 16:55:04.631797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631622 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5d91cb-f7ce-4209-9057-c77110345a8a" containerName="maas-api" Jul 22 16:55:04.631797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631628 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5d91cb-f7ce-4209-9057-c77110345a8a" containerName="maas-api" Jul 22 16:55:04.631797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631716 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5d91cb-f7ce-4209-9057-c77110345a8a" containerName="maas-api" Jul 22 16:55:04.631797 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.631726 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb87d48-bf9f-4fca-959d-fb23ebfc107c" containerName="revoke-keys" Jul 22 16:55:04.635225 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.635196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.637707 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.637684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b7a2b4dd-serving-cert\"" Jul 22 16:55:04.645893 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.645868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j"] Jul 22 16:55:04.673705 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.673670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm55r\" (UniqueName: \"kubernetes.io/projected/56e1be8e-6651-4d13-8a1c-0167d290feee-kube-api-access-tm55r\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.673870 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.673757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1be8e-6651-4d13-8a1c-0167d290feee-openshift-service-ca\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.673870 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.673786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.774630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.774592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm55r\" (UniqueName: \"kubernetes.io/projected/56e1be8e-6651-4d13-8a1c-0167d290feee-kube-api-access-tm55r\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.774630 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.774637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1be8e-6651-4d13-8a1c-0167d290feee-openshift-service-ca\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.774861 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.774654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.774861 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:55:04.774765 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-serving-cert: secret "maas-api-e2e-mt-b7a2b4dd-serving-cert" not found Jul 22 16:55:04.774861 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:55:04.774826 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls podName:56e1be8e-6651-4d13-8a1c-0167d290feee nodeName:}" failed. No retries permitted until 2026-07-22 16:55:05.274808222 +0000 UTC m=+2062.854902101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls") pod "maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" (UID: "56e1be8e-6651-4d13-8a1c-0167d290feee") : secret "maas-api-e2e-mt-b7a2b4dd-serving-cert" not found Jul 22 16:55:04.775281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.775259 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1be8e-6651-4d13-8a1c-0167d290feee-openshift-service-ca\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:04.783902 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:04.783872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm55r\" (UniqueName: \"kubernetes.io/projected/56e1be8e-6651-4d13-8a1c-0167d290feee-kube-api-access-tm55r\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:05.280863 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:05.280816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:05.283273 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:05.283238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls\") pod \"maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:05.550277 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:05.550151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:05.699757 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:05.699730 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j"] Jul 22 16:55:05.701263 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:55:05.701233 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e1be8e_6651_4d13_8a1c_0167d290feee.slice/crio-1be12d8f820833405664ac2d26230106eddb4a01016d7328ad70b539e5245c9d WatchSource:0}: Error finding container 1be12d8f820833405664ac2d26230106eddb4a01016d7328ad70b539e5245c9d: Status 404 returned error can't find the container with id 1be12d8f820833405664ac2d26230106eddb4a01016d7328ad70b539e5245c9d Jul 22 16:55:05.735056 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:05.735010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" event={"ID":"56e1be8e-6651-4d13-8a1c-0167d290feee","Type":"ContainerStarted","Data":"1be12d8f820833405664ac2d26230106eddb4a01016d7328ad70b539e5245c9d"} Jul 22 16:55:06.744033 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:06.743997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" event={"ID":"56e1be8e-6651-4d13-8a1c-0167d290feee","Type":"ContainerStarted","Data":"5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a"} Jul 22 16:55:06.744535 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:06.744191 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:06.764549 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:06.764492 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" podStartSLOduration=2.336519905 podStartE2EDuration="2.764472477s" totalImagesPullingTime="427.952572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:55:06.760902123 +0000 UTC m=+2064.340996021" watchObservedRunningTime="2026-07-22 16:55:06.764472477 +0000 UTC m=+2064.344566374" Jul 22 16:55:12.757601 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:12.757569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:16.811015 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.810978 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj"] Jul 22 16:55:16.815108 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.815091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:16.817659 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.817638 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xmwmv\"" Jul 22 16:55:16.823471 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.823446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj"] Jul 22 16:55:16.897578 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.897543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj\" (UID: \"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:16.998159 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.998100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj\" (UID: \"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:16.998705 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:16.998688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj\" (UID: \"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:17.127235 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:17.127158 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:17.272547 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:17.272515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj"] Jul 22 16:55:17.273901 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:55:17.273871 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a9d237_e4d9_49ce_a9c9_680e34a7d6d4.slice/crio-0e2a4bbe5a10c5e56b67aa3891e41f3a1832d5178c4508d65d39439fabb828fd WatchSource:0}: Error finding container 0e2a4bbe5a10c5e56b67aa3891e41f3a1832d5178c4508d65d39439fabb828fd: Status 404 returned error can't find the container with id 0e2a4bbe5a10c5e56b67aa3891e41f3a1832d5178c4508d65d39439fabb828fd Jul 22 16:55:17.808842 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:17.808733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" event={"ID":"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4","Type":"ContainerStarted","Data":"4422ab16ca0cf1660d83cfee6694376be2ed17b02ece8691f905b9b6a097dfed"} Jul 22 16:55:17.808842 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:17.808800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" event={"ID":"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4","Type":"ContainerStarted","Data":"0e2a4bbe5a10c5e56b67aa3891e41f3a1832d5178c4508d65d39439fabb828fd"} Jul 22 16:55:17.827502 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:17.827458 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" podStartSLOduration=1.827444888 podStartE2EDuration="1.827444888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:55:17.824554707 +0000 UTC m=+2075.404648630" watchObservedRunningTime="2026-07-22 16:55:17.827444888 +0000 UTC m=+2075.407538785" Jul 22 16:55:18.816383 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:18.816345 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4" containerID="4422ab16ca0cf1660d83cfee6694376be2ed17b02ece8691f905b9b6a097dfed" exitCode=0 Jul 22 16:55:18.816571 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:18.816434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" event={"ID":"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4","Type":"ContainerDied","Data":"4422ab16ca0cf1660d83cfee6694376be2ed17b02ece8691f905b9b6a097dfed"} Jul 22 16:55:19.960857 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:19.960835 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:20.023018 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:20.022979 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4-maas-api-service-ca\") pod \"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4\" (UID: \"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4\") " Jul 22 16:55:20.023337 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:20.023313 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4" (UID: "f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:55:20.124079 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:20.124007 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4-maas-api-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:55:20.830792 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:20.830760 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" Jul 22 16:55:20.830977 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:20.830798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj" event={"ID":"f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4","Type":"ContainerDied","Data":"0e2a4bbe5a10c5e56b67aa3891e41f3a1832d5178c4508d65d39439fabb828fd"} Jul 22 16:55:20.830977 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:20.830846 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e2a4bbe5a10c5e56b67aa3891e41f3a1832d5178c4508d65d39439fabb828fd" Jul 22 16:55:27.272901 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.272861 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j"] Jul 22 16:55:27.273374 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.273147 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" podUID="56e1be8e-6651-4d13-8a1c-0167d290feee" containerName="maas-api" containerID="cri-o://5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a" gracePeriod=30 Jul 22 16:55:27.528806 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.528739 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:27.699562 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.699519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm55r\" (UniqueName: \"kubernetes.io/projected/56e1be8e-6651-4d13-8a1c-0167d290feee-kube-api-access-tm55r\") pod \"56e1be8e-6651-4d13-8a1c-0167d290feee\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " Jul 22 16:55:27.699761 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.699595 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls\") pod \"56e1be8e-6651-4d13-8a1c-0167d290feee\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " Jul 22 16:55:27.699761 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.699721 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1be8e-6651-4d13-8a1c-0167d290feee-openshift-service-ca\") pod \"56e1be8e-6651-4d13-8a1c-0167d290feee\" (UID: \"56e1be8e-6651-4d13-8a1c-0167d290feee\") " Jul 22 16:55:27.700084 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.700060 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e1be8e-6651-4d13-8a1c-0167d290feee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56e1be8e-6651-4d13-8a1c-0167d290feee" (UID: "56e1be8e-6651-4d13-8a1c-0167d290feee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:55:27.701714 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.701692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56e1be8e-6651-4d13-8a1c-0167d290feee" (UID: "56e1be8e-6651-4d13-8a1c-0167d290feee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:55:27.701794 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.701749 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e1be8e-6651-4d13-8a1c-0167d290feee-kube-api-access-tm55r" (OuterVolumeSpecName: "kube-api-access-tm55r") pod "56e1be8e-6651-4d13-8a1c-0167d290feee" (UID: "56e1be8e-6651-4d13-8a1c-0167d290feee"). InnerVolumeSpecName "kube-api-access-tm55r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:55:27.801281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.801183 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm55r\" (UniqueName: \"kubernetes.io/projected/56e1be8e-6651-4d13-8a1c-0167d290feee-kube-api-access-tm55r\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:55:27.801281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.801226 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e1be8e-6651-4d13-8a1c-0167d290feee-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:55:27.801281 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.801238 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1be8e-6651-4d13-8a1c-0167d290feee-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:55:27.872766 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.872733 2580 generic.go:356] "Generic (PLEG): container finished" podID="56e1be8e-6651-4d13-8a1c-0167d290feee" containerID="5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a" exitCode=0 Jul 22 16:55:27.872936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.872795 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" Jul 22 16:55:27.872936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.872820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" event={"ID":"56e1be8e-6651-4d13-8a1c-0167d290feee","Type":"ContainerDied","Data":"5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a"} Jul 22 16:55:27.872936 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.872922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j" event={"ID":"56e1be8e-6651-4d13-8a1c-0167d290feee","Type":"ContainerDied","Data":"1be12d8f820833405664ac2d26230106eddb4a01016d7328ad70b539e5245c9d"} Jul 22 16:55:27.873119 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.872957 2580 scope.go:117] "RemoveContainer" containerID="5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a" Jul 22 16:55:27.884420 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.884386 2580 scope.go:117] "RemoveContainer" containerID="5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a" Jul 22 16:55:27.884668 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:55:27.884649 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a\": container with ID starting with 5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a not found: ID does not exist" containerID="5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a" Jul 22 16:55:27.884729 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.884675 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a"} err="failed to get container status \"5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a\": rpc error: code = NotFound desc = could not find container \"5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a\": container with ID starting with 5368223bc7120827fdd62ab636258d8c9f7de5fecc341052b984bd6b6ae9260a not found: ID does not exist" Jul 22 16:55:27.897174 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.897144 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j"] Jul 22 16:55:27.901429 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:27.901385 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b7a2b4dd-6b96f4789b-v7g6j"] Jul 22 16:55:28.980689 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:28.980657 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e1be8e-6651-4d13-8a1c-0167d290feee" path="/var/lib/kubelet/pods/56e1be8e-6651-4d13-8a1c-0167d290feee/volumes" Jul 22 16:55:31.915111 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:31.915072 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj"] Jul 22 16:55:31.918536 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:31.918498 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7a2b4dd-gxmxj"] Jul 22 16:55:32.980609 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:55:32.980575 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4" path="/var/lib/kubelet/pods/f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4/volumes" Jul 22 16:56:02.885417 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.885366 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4"] Jul 22 16:56:02.885926 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.885871 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4" containerName="revoke-keys" Jul 22 16:56:02.885926 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.885887 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4" containerName="revoke-keys" Jul 22 16:56:02.885926 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.885904 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e1be8e-6651-4d13-8a1c-0167d290feee" containerName="maas-api" Jul 22 16:56:02.885926 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.885909 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e1be8e-6651-4d13-8a1c-0167d290feee" containerName="maas-api" Jul 22 16:56:02.886160 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.886002 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a9d237-e4d9-49ce-a9c9-680e34a7d6d4" containerName="revoke-keys" Jul 22 16:56:02.886160 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.886014 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e1be8e-6651-4d13-8a1c-0167d290feee" containerName="maas-api" Jul 22 16:56:02.889285 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.889264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:02.892527 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.892508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-db88bcf6-serving-cert\"" Jul 22 16:56:02.902935 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:02.902909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4"] Jul 22 16:56:03.035683 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.035642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079b768e-d3b9-4220-bd71-072d18c42482-openshift-service-ca\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.035898 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.035723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079b768e-d3b9-4220-bd71-072d18c42482-maas-api-tls\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.035898 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.035755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt4sv\" (UniqueName: \"kubernetes.io/projected/079b768e-d3b9-4220-bd71-072d18c42482-kube-api-access-pt4sv\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.136974 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.136819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079b768e-d3b9-4220-bd71-072d18c42482-openshift-service-ca\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.136974 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.136873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079b768e-d3b9-4220-bd71-072d18c42482-maas-api-tls\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.136974 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.136900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt4sv\" (UniqueName: \"kubernetes.io/projected/079b768e-d3b9-4220-bd71-072d18c42482-kube-api-access-pt4sv\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.137623 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.137601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079b768e-d3b9-4220-bd71-072d18c42482-openshift-service-ca\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.139570 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.139545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079b768e-d3b9-4220-bd71-072d18c42482-maas-api-tls\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.145930 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.145904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt4sv\" (UniqueName: \"kubernetes.io/projected/079b768e-d3b9-4220-bd71-072d18c42482-kube-api-access-pt4sv\") pod \"maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.199869 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.199829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:03.367222 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.367196 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4"] Jul 22 16:56:03.369145 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:56:03.369119 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079b768e_d3b9_4220_bd71_072d18c42482.slice/crio-ea1827a10f3b5b9fab42913fc8e5627a831340a1ea88b17585cd6bd558aa3249 WatchSource:0}: Error finding container ea1827a10f3b5b9fab42913fc8e5627a831340a1ea88b17585cd6bd558aa3249: Status 404 returned error can't find the container with id ea1827a10f3b5b9fab42913fc8e5627a831340a1ea88b17585cd6bd558aa3249 Jul 22 16:56:03.370559 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:03.370541 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:56:04.089581 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:04.089544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" event={"ID":"079b768e-d3b9-4220-bd71-072d18c42482","Type":"ContainerStarted","Data":"af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645"} Jul 22 16:56:04.090050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:04.089600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" event={"ID":"079b768e-d3b9-4220-bd71-072d18c42482","Type":"ContainerStarted","Data":"ea1827a10f3b5b9fab42913fc8e5627a831340a1ea88b17585cd6bd558aa3249"} Jul 22 16:56:04.090050 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:04.089696 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:56:04.108145 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:04.108080 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" podStartSLOduration=1.7820059320000001 podStartE2EDuration="2.108063126s" totalImagesPullingTime="326.057194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:56:04.106895924 +0000 UTC m=+2121.686989820" watchObservedRunningTime="2026-07-22 16:56:04.108063126 +0000 UTC m=+2121.688157022" Jul 22 16:56:10.101052 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:56:10.101023 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:57:07.006358 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.006317 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4"] Jul 22 16:57:07.007292 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.006685 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" podUID="079b768e-d3b9-4220-bd71-072d18c42482" containerName="maas-api" containerID="cri-o://af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645" gracePeriod=30 Jul 22 16:57:07.264256 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.264198 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:57:07.349493 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.349458 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079b768e-d3b9-4220-bd71-072d18c42482-openshift-service-ca\") pod \"079b768e-d3b9-4220-bd71-072d18c42482\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " Jul 22 16:57:07.349679 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.349532 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079b768e-d3b9-4220-bd71-072d18c42482-maas-api-tls\") pod \"079b768e-d3b9-4220-bd71-072d18c42482\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " Jul 22 16:57:07.349679 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.349583 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt4sv\" (UniqueName: \"kubernetes.io/projected/079b768e-d3b9-4220-bd71-072d18c42482-kube-api-access-pt4sv\") pod \"079b768e-d3b9-4220-bd71-072d18c42482\" (UID: \"079b768e-d3b9-4220-bd71-072d18c42482\") " Jul 22 16:57:07.349919 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.349883 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079b768e-d3b9-4220-bd71-072d18c42482-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "079b768e-d3b9-4220-bd71-072d18c42482" (UID: "079b768e-d3b9-4220-bd71-072d18c42482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:57:07.351635 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.351606 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079b768e-d3b9-4220-bd71-072d18c42482-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "079b768e-d3b9-4220-bd71-072d18c42482" (UID: "079b768e-d3b9-4220-bd71-072d18c42482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:57:07.351748 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.351685 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079b768e-d3b9-4220-bd71-072d18c42482-kube-api-access-pt4sv" (OuterVolumeSpecName: "kube-api-access-pt4sv") pod "079b768e-d3b9-4220-bd71-072d18c42482" (UID: "079b768e-d3b9-4220-bd71-072d18c42482"). InnerVolumeSpecName "kube-api-access-pt4sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:57:07.450476 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.450434 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079b768e-d3b9-4220-bd71-072d18c42482-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:57:07.450476 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.450463 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079b768e-d3b9-4220-bd71-072d18c42482-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:57:07.450476 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.450473 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt4sv\" (UniqueName: \"kubernetes.io/projected/079b768e-d3b9-4220-bd71-072d18c42482-kube-api-access-pt4sv\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 16:57:07.482519 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.482485 2580 generic.go:356] "Generic (PLEG): container finished" podID="079b768e-d3b9-4220-bd71-072d18c42482" containerID="af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645" exitCode=0 Jul 22 16:57:07.482693 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.482537 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" Jul 22 16:57:07.482693 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.482565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" event={"ID":"079b768e-d3b9-4220-bd71-072d18c42482","Type":"ContainerDied","Data":"af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645"} Jul 22 16:57:07.482693 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.482607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4" event={"ID":"079b768e-d3b9-4220-bd71-072d18c42482","Type":"ContainerDied","Data":"ea1827a10f3b5b9fab42913fc8e5627a831340a1ea88b17585cd6bd558aa3249"} Jul 22 16:57:07.482693 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.482628 2580 scope.go:117] "RemoveContainer" containerID="af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645" Jul 22 16:57:07.492247 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.492227 2580 scope.go:117] "RemoveContainer" containerID="af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645" Jul 22 16:57:07.492598 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:57:07.492574 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645\": container with ID starting with af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645 not found: ID does not exist" containerID="af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645" Jul 22 16:57:07.492682 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.492610 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645"} err="failed to get container status \"af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645\": rpc error: code = NotFound desc = could not find container \"af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645\": container with ID starting with af872ec7629376bdf241ca3d3d3affc738860f0d661d734e96dc5464e4d12645 not found: ID does not exist" Jul 22 16:57:07.507103 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.507073 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4"] Jul 22 16:57:07.511742 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:07.511718 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db88bcf6-5fc686c655-d69q4"] Jul 22 16:57:08.980366 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:08.980327 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079b768e-d3b9-4220-bd71-072d18c42482" path="/var/lib/kubelet/pods/079b768e-d3b9-4220-bd71-072d18c42482/volumes" Jul 22 16:57:43.247774 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:57:43.247742 2580 scope.go:117] "RemoveContainer" containerID="ad9b5d91b24f67435d70aff5ea54c3f8eaa5403c8e1605989a1d7ed665da938b" Jul 22 16:59:03.783608 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.783577 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s"] Jul 22 16:59:03.784073 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.784036 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079b768e-d3b9-4220-bd71-072d18c42482" containerName="maas-api" Jul 22 16:59:03.784073 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.784048 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="079b768e-d3b9-4220-bd71-072d18c42482" containerName="maas-api" Jul 22 16:59:03.784150 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.784131 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="079b768e-d3b9-4220-bd71-072d18c42482" containerName="maas-api" Jul 22 16:59:03.787325 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.787309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.789792 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.789769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-314e2c-serving-cert\"" Jul 22 16:59:03.799262 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.799239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s"] Jul 22 16:59:03.854524 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.854475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaff5f32-02e0-4183-aba2-9a5d1722dc80-maas-api-tls\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.854708 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.854605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ndb\" (UniqueName: \"kubernetes.io/projected/aaff5f32-02e0-4183-aba2-9a5d1722dc80-kube-api-access-p8ndb\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.854793 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.854709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaff5f32-02e0-4183-aba2-9a5d1722dc80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.956017 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.955977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ndb\" (UniqueName: \"kubernetes.io/projected/aaff5f32-02e0-4183-aba2-9a5d1722dc80-kube-api-access-p8ndb\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.956181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.956034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaff5f32-02e0-4183-aba2-9a5d1722dc80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.956181 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.956073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaff5f32-02e0-4183-aba2-9a5d1722dc80-maas-api-tls\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.956785 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.956752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaff5f32-02e0-4183-aba2-9a5d1722dc80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.958525 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.958506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaff5f32-02e0-4183-aba2-9a5d1722dc80-maas-api-tls\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:03.965216 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:03.965192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ndb\" (UniqueName: \"kubernetes.io/projected/aaff5f32-02e0-4183-aba2-9a5d1722dc80-kube-api-access-p8ndb\") pod \"maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:04.099676 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:04.099585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:04.445683 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:04.445657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s"] Jul 22 16:59:04.448238 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:59:04.448196 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaff5f32_02e0_4183_aba2_9a5d1722dc80.slice/crio-a2c988e7fde7bfe4bd6a717b1be88fecaf0ba33127dcb59565b80a6798a64858 WatchSource:0}: Error finding container a2c988e7fde7bfe4bd6a717b1be88fecaf0ba33127dcb59565b80a6798a64858: Status 404 returned error can't find the container with id a2c988e7fde7bfe4bd6a717b1be88fecaf0ba33127dcb59565b80a6798a64858 Jul 22 16:59:05.210049 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:05.210010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" event={"ID":"aaff5f32-02e0-4183-aba2-9a5d1722dc80","Type":"ContainerStarted","Data":"701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc"} Jul 22 16:59:05.210530 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:05.210095 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" event={"ID":"aaff5f32-02e0-4183-aba2-9a5d1722dc80","Type":"ContainerStarted","Data":"a2c988e7fde7bfe4bd6a717b1be88fecaf0ba33127dcb59565b80a6798a64858"} Jul 22 16:59:05.210530 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:05.210214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:05.227042 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:05.226991 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" podStartSLOduration=1.678624318 podStartE2EDuration="2.226978016s" totalImagesPullingTime="548.353698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:59:05.226079454 +0000 UTC m=+2302.806173351" watchObservedRunningTime="2026-07-22 16:59:05.226978016 +0000 UTC m=+2302.807071912" Jul 22 16:59:11.223045 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:11.223014 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 16:59:11.901921 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:11.901884 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd"] Jul 22 16:59:11.905830 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:11.905808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:11.907889 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:11.907864 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4b1cd3-serving-cert\"" Jul 22 16:59:11.919149 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:11.919123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd"] Jul 22 16:59:12.034443 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.034408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.034644 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.034598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726accac-c735-4af1-8ca8-d82974f8f497-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.034754 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.034721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5n4\" (UniqueName: \"kubernetes.io/projected/726accac-c735-4af1-8ca8-d82974f8f497-kube-api-access-5x5n4\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.136046 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.136005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5n4\" (UniqueName: \"kubernetes.io/projected/726accac-c735-4af1-8ca8-d82974f8f497-kube-api-access-5x5n4\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.136242 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.136129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.136242 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.136187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726accac-c735-4af1-8ca8-d82974f8f497-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.136370 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:59:12.136259 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-serving-cert: secret "maas-api-e2e-inf-b-4b1cd3-serving-cert" not found Jul 22 16:59:12.136370 ip-10-0-133-136 kubenswrapper[2580]: E0722 16:59:12.136356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls podName:726accac-c735-4af1-8ca8-d82974f8f497 nodeName:}" failed. No retries permitted until 2026-07-22 16:59:12.636322312 +0000 UTC m=+2310.216416205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls") pod "maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" (UID: "726accac-c735-4af1-8ca8-d82974f8f497") : secret "maas-api-e2e-inf-b-4b1cd3-serving-cert" not found Jul 22 16:59:12.136981 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.136960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726accac-c735-4af1-8ca8-d82974f8f497-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.146094 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.146071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5n4\" (UniqueName: \"kubernetes.io/projected/726accac-c735-4af1-8ca8-d82974f8f497-kube-api-access-5x5n4\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.642006 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.641962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.644564 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.644537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.817069 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.817032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:12.968958 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:12.968933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd"] Jul 22 16:59:12.971070 ip-10-0-133-136 kubenswrapper[2580]: W0722 16:59:12.971036 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726accac_c735_4af1_8ca8_d82974f8f497.slice/crio-44bc4a5e5fd3dadaf70553c7f46e4408c04a358010811971929f180e6fff3d46 WatchSource:0}: Error finding container 44bc4a5e5fd3dadaf70553c7f46e4408c04a358010811971929f180e6fff3d46: Status 404 returned error can't find the container with id 44bc4a5e5fd3dadaf70553c7f46e4408c04a358010811971929f180e6fff3d46 Jul 22 16:59:13.264817 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:13.264780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" event={"ID":"726accac-c735-4af1-8ca8-d82974f8f497","Type":"ContainerStarted","Data":"44bc4a5e5fd3dadaf70553c7f46e4408c04a358010811971929f180e6fff3d46"} Jul 22 16:59:14.272778 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:14.272743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" event={"ID":"726accac-c735-4af1-8ca8-d82974f8f497","Type":"ContainerStarted","Data":"1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35"} Jul 22 16:59:14.273200 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:14.272885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:14.296116 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:14.296068 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" podStartSLOduration=2.914744546 podStartE2EDuration="3.296053569s" totalImagesPullingTime="381.309023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:59:14.293648724 +0000 UTC m=+2311.873742620" watchObservedRunningTime="2026-07-22 16:59:14.296053569 +0000 UTC m=+2311.876147465" Jul 22 16:59:20.284168 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:20.284137 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 16:59:43.282969 ip-10-0-133-136 kubenswrapper[2580]: I0722 16:59:43.282939 2580 scope.go:117] "RemoveContainer" containerID="717246040ae1557cf2edfdc5de5efa5004678b79145bdbfec718f29fb01c2241" Jul 22 17:00:00.131093 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.131060 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl"] Jul 22 17:00:00.134916 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.134890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" Jul 22 17:00:00.138314 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.138278 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xmwmv\"" Jul 22 17:00:00.145175 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.145111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl"] Jul 22 17:00:00.158810 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.158782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" Jul 22 17:00:00.304329 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.304303 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl"] Jul 22 17:00:00.307036 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:00:00.307007 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee9a43a_b6e5_41e1_af8d_d12ee8154754.slice/crio-db71e260b3e13b0dc1e2b8d12be2c7b20162309b7645bd58cb7161b19ea41ae7 WatchSource:0}: Error finding container db71e260b3e13b0dc1e2b8d12be2c7b20162309b7645bd58cb7161b19ea41ae7: Status 404 returned error can't find the container with id db71e260b3e13b0dc1e2b8d12be2c7b20162309b7645bd58cb7161b19ea41ae7 Jul 22 17:00:00.567478 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:00.567440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" event={"ID":"fee9a43a-b6e5-41e1-af8d-d12ee8154754","Type":"ContainerStarted","Data":"db71e260b3e13b0dc1e2b8d12be2c7b20162309b7645bd58cb7161b19ea41ae7"} Jul 22 17:00:01.575221 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:01.575186 2580 generic.go:356] "Generic (PLEG): container finished" podID="fee9a43a-b6e5-41e1-af8d-d12ee8154754" containerID="ca0995dfac7cae8038a1a4c09fabb57e0959594862d440cc4fe0be6f502e2912" exitCode=0 Jul 22 17:00:01.575696 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:01.575278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" event={"ID":"fee9a43a-b6e5-41e1-af8d-d12ee8154754","Type":"ContainerDied","Data":"ca0995dfac7cae8038a1a4c09fabb57e0959594862d440cc4fe0be6f502e2912"} Jul 22 17:00:02.719851 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:02.719826 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" Jul 22 17:00:03.587879 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:03.587845 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" Jul 22 17:00:03.588092 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:03.587878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl" event={"ID":"fee9a43a-b6e5-41e1-af8d-d12ee8154754","Type":"ContainerDied","Data":"db71e260b3e13b0dc1e2b8d12be2c7b20162309b7645bd58cb7161b19ea41ae7"} Jul 22 17:00:03.588092 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:03.587930 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db71e260b3e13b0dc1e2b8d12be2c7b20162309b7645bd58cb7161b19ea41ae7" Jul 22 17:00:48.176826 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.176284 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s"] Jul 22 17:00:48.176826 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.176593 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" podUID="aaff5f32-02e0-4183-aba2-9a5d1722dc80" containerName="maas-api" containerID="cri-o://701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc" gracePeriod=30 Jul 22 17:00:48.448282 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.448251 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 17:00:48.501985 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.501939 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8ndb\" (UniqueName: \"kubernetes.io/projected/aaff5f32-02e0-4183-aba2-9a5d1722dc80-kube-api-access-p8ndb\") pod \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " Jul 22 17:00:48.502181 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.502047 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaff5f32-02e0-4183-aba2-9a5d1722dc80-openshift-service-ca\") pod \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " Jul 22 17:00:48.502181 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.502084 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaff5f32-02e0-4183-aba2-9a5d1722dc80-maas-api-tls\") pod \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\" (UID: \"aaff5f32-02e0-4183-aba2-9a5d1722dc80\") " Jul 22 17:00:48.502512 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.502487 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaff5f32-02e0-4183-aba2-9a5d1722dc80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aaff5f32-02e0-4183-aba2-9a5d1722dc80" (UID: "aaff5f32-02e0-4183-aba2-9a5d1722dc80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:00:48.504196 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.504169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaff5f32-02e0-4183-aba2-9a5d1722dc80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aaff5f32-02e0-4183-aba2-9a5d1722dc80" (UID: "aaff5f32-02e0-4183-aba2-9a5d1722dc80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:00:48.504275 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.504188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaff5f32-02e0-4183-aba2-9a5d1722dc80-kube-api-access-p8ndb" (OuterVolumeSpecName: "kube-api-access-p8ndb") pod "aaff5f32-02e0-4183-aba2-9a5d1722dc80" (UID: "aaff5f32-02e0-4183-aba2-9a5d1722dc80"). InnerVolumeSpecName "kube-api-access-p8ndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:00:48.603184 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.603138 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaff5f32-02e0-4183-aba2-9a5d1722dc80-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:00:48.603184 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.603186 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ndb\" (UniqueName: \"kubernetes.io/projected/aaff5f32-02e0-4183-aba2-9a5d1722dc80-kube-api-access-p8ndb\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:00:48.603458 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.603201 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaff5f32-02e0-4183-aba2-9a5d1722dc80-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:00:48.874039 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.874000 2580 generic.go:356] "Generic (PLEG): container finished" podID="aaff5f32-02e0-4183-aba2-9a5d1722dc80" containerID="701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc" exitCode=0 Jul 22 17:00:48.874211 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.874048 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" Jul 22 17:00:48.874211 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.874089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" event={"ID":"aaff5f32-02e0-4183-aba2-9a5d1722dc80","Type":"ContainerDied","Data":"701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc"} Jul 22 17:00:48.874211 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.874149 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s" event={"ID":"aaff5f32-02e0-4183-aba2-9a5d1722dc80","Type":"ContainerDied","Data":"a2c988e7fde7bfe4bd6a717b1be88fecaf0ba33127dcb59565b80a6798a64858"} Jul 22 17:00:48.874211 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.874174 2580 scope.go:117] "RemoveContainer" containerID="701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc" Jul 22 17:00:48.884929 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.884908 2580 scope.go:117] "RemoveContainer" containerID="701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc" Jul 22 17:00:48.885271 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:00:48.885248 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc\": container with ID starting with 701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc not found: ID does not exist" containerID="701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc" Jul 22 17:00:48.885385 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.885280 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc"} err="failed to get container status \"701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc\": rpc error: code = NotFound desc = could not find container \"701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc\": container with ID starting with 701d0f22a001d1171c57b899b3ee3ebd3f56c3b1d87ace6e4cce0ddf747138fc not found: ID does not exist" Jul 22 17:00:48.898700 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.898664 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s"] Jul 22 17:00:48.903018 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.902979 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-314e2c-58bdf87c4-jc57s"] Jul 22 17:00:48.980958 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:00:48.980919 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaff5f32-02e0-4183-aba2-9a5d1722dc80" path="/var/lib/kubelet/pods/aaff5f32-02e0-4183-aba2-9a5d1722dc80/volumes" Jul 22 17:01:00.066768 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.066726 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48"] Jul 22 17:01:00.067263 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.067188 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee9a43a-b6e5-41e1-af8d-d12ee8154754" containerName="cleanup" Jul 22 17:01:00.067263 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.067199 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee9a43a-b6e5-41e1-af8d-d12ee8154754" containerName="cleanup" Jul 22 17:01:00.067263 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.067223 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaff5f32-02e0-4183-aba2-9a5d1722dc80" containerName="maas-api" Jul 22 17:01:00.067263 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.067229 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaff5f32-02e0-4183-aba2-9a5d1722dc80" containerName="maas-api" Jul 22 17:01:00.067449 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.067300 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaff5f32-02e0-4183-aba2-9a5d1722dc80" containerName="maas-api" Jul 22 17:01:00.067449 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.067309 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee9a43a-b6e5-41e1-af8d-d12ee8154754" containerName="cleanup" Jul 22 17:01:00.070621 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.070599 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:00.073042 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.073022 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xmwmv\"" Jul 22 17:01:00.081901 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.081871 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48"] Jul 22 17:01:00.218463 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.218366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca223d-498a-46fb-afe5-df50fc3bfc74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48\" (UID: \"52ca223d-498a-46fb-afe5-df50fc3bfc74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:00.319378 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.319273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca223d-498a-46fb-afe5-df50fc3bfc74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48\" (UID: \"52ca223d-498a-46fb-afe5-df50fc3bfc74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:00.319970 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.319945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca223d-498a-46fb-afe5-df50fc3bfc74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48\" (UID: \"52ca223d-498a-46fb-afe5-df50fc3bfc74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:00.381932 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.381891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:00.550246 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.550215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48"] Jul 22 17:01:00.552239 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:01:00.552207 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ca223d_498a_46fb_afe5_df50fc3bfc74.slice/crio-f91c391d58c753c355e3397dd8533ffe20fe1839636281ee2ecb0f56ff364b41 WatchSource:0}: Error finding container f91c391d58c753c355e3397dd8533ffe20fe1839636281ee2ecb0f56ff364b41: Status 404 returned error can't find the container with id f91c391d58c753c355e3397dd8533ffe20fe1839636281ee2ecb0f56ff364b41 Jul 22 17:01:00.953851 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.953810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" event={"ID":"52ca223d-498a-46fb-afe5-df50fc3bfc74","Type":"ContainerStarted","Data":"3552588634d292db26bf3c9f5971cf78bcf2cb8a2ccd094132433803a295f8b4"} Jul 22 17:01:00.954110 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.953896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" event={"ID":"52ca223d-498a-46fb-afe5-df50fc3bfc74","Type":"ContainerStarted","Data":"f91c391d58c753c355e3397dd8533ffe20fe1839636281ee2ecb0f56ff364b41"} Jul 22 17:01:00.976254 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:00.976200 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" podStartSLOduration=0.976184099 podStartE2EDuration="976.184099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:01:00.972746574 +0000 UTC m=+2418.552840493" watchObservedRunningTime="2026-07-22 17:01:00.976184099 +0000 UTC m=+2418.556277996" Jul 22 17:01:01.960982 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:01.960942 2580 generic.go:356] "Generic (PLEG): container finished" podID="52ca223d-498a-46fb-afe5-df50fc3bfc74" containerID="3552588634d292db26bf3c9f5971cf78bcf2cb8a2ccd094132433803a295f8b4" exitCode=0 Jul 22 17:01:01.961373 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:01.961034 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" event={"ID":"52ca223d-498a-46fb-afe5-df50fc3bfc74","Type":"ContainerDied","Data":"3552588634d292db26bf3c9f5971cf78bcf2cb8a2ccd094132433803a295f8b4"} Jul 22 17:01:03.122662 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.122626 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:03.251972 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.251934 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca223d-498a-46fb-afe5-df50fc3bfc74-maas-api-service-ca\") pod \"52ca223d-498a-46fb-afe5-df50fc3bfc74\" (UID: \"52ca223d-498a-46fb-afe5-df50fc3bfc74\") " Jul 22 17:01:03.252316 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.252291 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ca223d-498a-46fb-afe5-df50fc3bfc74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52ca223d-498a-46fb-afe5-df50fc3bfc74" (UID: "52ca223d-498a-46fb-afe5-df50fc3bfc74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:01:03.353191 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.353148 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca223d-498a-46fb-afe5-df50fc3bfc74-maas-api-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:01:03.975645 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.975614 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" Jul 22 17:01:03.975834 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.975651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48" event={"ID":"52ca223d-498a-46fb-afe5-df50fc3bfc74","Type":"ContainerDied","Data":"f91c391d58c753c355e3397dd8533ffe20fe1839636281ee2ecb0f56ff364b41"} Jul 22 17:01:03.975834 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:03.975696 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91c391d58c753c355e3397dd8533ffe20fe1839636281ee2ecb0f56ff364b41" Jul 22 17:01:10.534853 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.534758 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd"] Jul 22 17:01:10.535289 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.535022 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" podUID="726accac-c735-4af1-8ca8-d82974f8f497" containerName="maas-api" containerID="cri-o://1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35" gracePeriod=30 Jul 22 17:01:10.562599 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.562563 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl"] Jul 22 17:01:10.565188 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.565163 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b1cd3-29745660-6vkxl"] Jul 22 17:01:10.793062 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.792979 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 17:01:10.927379 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.927337 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls\") pod \"726accac-c735-4af1-8ca8-d82974f8f497\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " Jul 22 17:01:10.927620 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.927428 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726accac-c735-4af1-8ca8-d82974f8f497-openshift-service-ca\") pod \"726accac-c735-4af1-8ca8-d82974f8f497\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " Jul 22 17:01:10.927620 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.927495 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x5n4\" (UniqueName: \"kubernetes.io/projected/726accac-c735-4af1-8ca8-d82974f8f497-kube-api-access-5x5n4\") pod \"726accac-c735-4af1-8ca8-d82974f8f497\" (UID: \"726accac-c735-4af1-8ca8-d82974f8f497\") " Jul 22 17:01:10.927794 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.927765 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726accac-c735-4af1-8ca8-d82974f8f497-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "726accac-c735-4af1-8ca8-d82974f8f497" (UID: "726accac-c735-4af1-8ca8-d82974f8f497"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:01:10.927926 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.927914 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726accac-c735-4af1-8ca8-d82974f8f497-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:01:10.929613 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.929590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "726accac-c735-4af1-8ca8-d82974f8f497" (UID: "726accac-c735-4af1-8ca8-d82974f8f497"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:01:10.929733 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.929640 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726accac-c735-4af1-8ca8-d82974f8f497-kube-api-access-5x5n4" (OuterVolumeSpecName: "kube-api-access-5x5n4") pod "726accac-c735-4af1-8ca8-d82974f8f497" (UID: "726accac-c735-4af1-8ca8-d82974f8f497"). InnerVolumeSpecName "kube-api-access-5x5n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:01:10.981020 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:10.980982 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee9a43a-b6e5-41e1-af8d-d12ee8154754" path="/var/lib/kubelet/pods/fee9a43a-b6e5-41e1-af8d-d12ee8154754/volumes" Jul 22 17:01:11.020902 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.020859 2580 generic.go:356] "Generic (PLEG): container finished" podID="726accac-c735-4af1-8ca8-d82974f8f497" containerID="1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35" exitCode=0 Jul 22 17:01:11.020902 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.020909 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" Jul 22 17:01:11.021111 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.020950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" event={"ID":"726accac-c735-4af1-8ca8-d82974f8f497","Type":"ContainerDied","Data":"1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35"} Jul 22 17:01:11.021111 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.021010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd" event={"ID":"726accac-c735-4af1-8ca8-d82974f8f497","Type":"ContainerDied","Data":"44bc4a5e5fd3dadaf70553c7f46e4408c04a358010811971929f180e6fff3d46"} Jul 22 17:01:11.021111 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.021037 2580 scope.go:117] "RemoveContainer" containerID="1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35" Jul 22 17:01:11.028646 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.028616 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5n4\" (UniqueName: \"kubernetes.io/projected/726accac-c735-4af1-8ca8-d82974f8f497-kube-api-access-5x5n4\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:01:11.028783 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.028645 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726accac-c735-4af1-8ca8-d82974f8f497-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:01:11.035133 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.035102 2580 scope.go:117] "RemoveContainer" containerID="1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35" Jul 22 17:01:11.035559 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:01:11.035522 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35\": container with ID starting with 1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35 not found: ID does not exist" containerID="1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35" Jul 22 17:01:11.035671 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.035572 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35"} err="failed to get container status \"1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35\": rpc error: code = NotFound desc = could not find container \"1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35\": container with ID starting with 1dd418ff461b95505406e0e0789a56351dd25d37a75faf60c210c93c7c4fbf35 not found: ID does not exist" Jul 22 17:01:11.041618 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.041584 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd"] Jul 22 17:01:11.047591 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:11.047500 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b1cd3-644bc66878-mpkfd"] Jul 22 17:01:12.981901 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:12.981864 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726accac-c735-4af1-8ca8-d82974f8f497" path="/var/lib/kubelet/pods/726accac-c735-4af1-8ca8-d82974f8f497/volumes" Jul 22 17:01:15.159366 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:15.159318 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48"] Jul 22 17:01:15.164699 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:15.164661 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b1cd3-jgn48"] Jul 22 17:01:16.980302 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:16.980264 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ca223d-498a-46fb-afe5-df50fc3bfc74" path="/var/lib/kubelet/pods/52ca223d-498a-46fb-afe5-df50fc3bfc74/volumes" Jul 22 17:01:26.317712 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.317654 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv"] Jul 22 17:01:26.319378 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.319346 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726accac-c735-4af1-8ca8-d82974f8f497" containerName="maas-api" Jul 22 17:01:26.319378 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.319377 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="726accac-c735-4af1-8ca8-d82974f8f497" containerName="maas-api" Jul 22 17:01:26.319597 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.319444 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ca223d-498a-46fb-afe5-df50fc3bfc74" containerName="revoke-keys" Jul 22 17:01:26.319597 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.319460 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ca223d-498a-46fb-afe5-df50fc3bfc74" containerName="revoke-keys" Jul 22 17:01:26.319698 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.319664 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ca223d-498a-46fb-afe5-df50fc3bfc74" containerName="revoke-keys" Jul 22 17:01:26.319698 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.319691 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="726accac-c735-4af1-8ca8-d82974f8f497" containerName="maas-api" Jul 22 17:01:26.324241 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.324211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.326666 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.326641 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bf7663-serving-cert\"" Jul 22 17:01:26.332526 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.332495 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv"] Jul 22 17:01:26.377668 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.377624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5qn\" (UniqueName: \"kubernetes.io/projected/87873f81-233d-45fb-8d28-b00ce410669c-kube-api-access-pf5qn\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.377668 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.377669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87873f81-233d-45fb-8d28-b00ce410669c-maas-api-tls\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.377892 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.377701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87873f81-233d-45fb-8d28-b00ce410669c-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.478963 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.478915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5qn\" (UniqueName: \"kubernetes.io/projected/87873f81-233d-45fb-8d28-b00ce410669c-kube-api-access-pf5qn\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.478963 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.478961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87873f81-233d-45fb-8d28-b00ce410669c-maas-api-tls\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.479239 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.478985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87873f81-233d-45fb-8d28-b00ce410669c-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.479776 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.479752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87873f81-233d-45fb-8d28-b00ce410669c-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.481471 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.481448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87873f81-233d-45fb-8d28-b00ce410669c-maas-api-tls\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.488950 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.488914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5qn\" (UniqueName: \"kubernetes.io/projected/87873f81-233d-45fb-8d28-b00ce410669c-kube-api-access-pf5qn\") pod \"maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.639183 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.639086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:26.794385 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.794356 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv"] Jul 22 17:01:26.795239 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:01:26.795211 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87873f81_233d_45fb_8d28_b00ce410669c.slice/crio-cc0b5d367ab0fe36d8c01f1a36c8cccb138f19572b91963d2b13c96512881ab3 WatchSource:0}: Error finding container cc0b5d367ab0fe36d8c01f1a36c8cccb138f19572b91963d2b13c96512881ab3: Status 404 returned error can't find the container with id cc0b5d367ab0fe36d8c01f1a36c8cccb138f19572b91963d2b13c96512881ab3 Jul 22 17:01:26.796682 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:26.796661 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:01:27.119804 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:27.119764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" event={"ID":"87873f81-233d-45fb-8d28-b00ce410669c","Type":"ContainerStarted","Data":"cc0b5d367ab0fe36d8c01f1a36c8cccb138f19572b91963d2b13c96512881ab3"} Jul 22 17:01:28.128679 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:28.128638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" event={"ID":"87873f81-233d-45fb-8d28-b00ce410669c","Type":"ContainerStarted","Data":"2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59"} Jul 22 17:01:28.129107 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:28.128764 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:28.149576 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:28.149520 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" podStartSLOduration=1.714353682 podStartE2EDuration="2.149502437s" totalImagesPullingTime="435.148755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:01:28.146148353 +0000 UTC m=+2445.726242250" watchObservedRunningTime="2026-07-22 17:01:28.149502437 +0000 UTC m=+2445.729596333" Jul 22 17:01:34.140176 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.140145 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:01:34.493689 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.493650 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds"] Jul 22 17:01:34.499028 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.498998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.501642 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.501617 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0f66aa-serving-cert\"" Jul 22 17:01:34.507960 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.507932 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds"] Jul 22 17:01:34.564334 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.564290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63678b91-7d74-4315-80f3-1d14245b89a9-maas-api-tls\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.564334 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.564330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63678b91-7d74-4315-80f3-1d14245b89a9-openshift-service-ca\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.564630 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.564354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxrh\" (UniqueName: \"kubernetes.io/projected/63678b91-7d74-4315-80f3-1d14245b89a9-kube-api-access-mzxrh\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.665732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.665694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63678b91-7d74-4315-80f3-1d14245b89a9-maas-api-tls\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.665732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.665733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63678b91-7d74-4315-80f3-1d14245b89a9-openshift-service-ca\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.665953 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.665753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxrh\" (UniqueName: \"kubernetes.io/projected/63678b91-7d74-4315-80f3-1d14245b89a9-kube-api-access-mzxrh\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.666675 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.666649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63678b91-7d74-4315-80f3-1d14245b89a9-openshift-service-ca\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.668300 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.668282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63678b91-7d74-4315-80f3-1d14245b89a9-maas-api-tls\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.678089 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.678054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxrh\" (UniqueName: \"kubernetes.io/projected/63678b91-7d74-4315-80f3-1d14245b89a9-kube-api-access-mzxrh\") pod \"maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.816263 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.816153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:34.978626 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:01:34.978580 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63678b91_7d74_4315_80f3_1d14245b89a9.slice/crio-e0be4b8c699819c33832a99d528999e8dfcffabd493b81dddbeb71ce12ee2fa0 WatchSource:0}: Error finding container e0be4b8c699819c33832a99d528999e8dfcffabd493b81dddbeb71ce12ee2fa0: Status 404 returned error can't find the container with id e0be4b8c699819c33832a99d528999e8dfcffabd493b81dddbeb71ce12ee2fa0 Jul 22 17:01:34.983315 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:34.983293 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds"] Jul 22 17:01:35.178480 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:35.178360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" event={"ID":"63678b91-7d74-4315-80f3-1d14245b89a9","Type":"ContainerStarted","Data":"e0be4b8c699819c33832a99d528999e8dfcffabd493b81dddbeb71ce12ee2fa0"} Jul 22 17:01:36.186663 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:36.186628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" event={"ID":"63678b91-7d74-4315-80f3-1d14245b89a9","Type":"ContainerStarted","Data":"ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6"} Jul 22 17:01:36.187097 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:36.186750 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:36.215803 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:36.215750 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" podStartSLOduration=1.703860312 podStartE2EDuration="2.215729428s" totalImagesPullingTime="511.869116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:01:36.213065444 +0000 UTC m=+2453.793159341" watchObservedRunningTime="2026-07-22 17:01:36.215729428 +0000 UTC m=+2453.795823373" Jul 22 17:01:42.198886 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:42.198849 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:01:43.314896 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:01:43.314855 2580 scope.go:117] "RemoveContainer" containerID="4422ab16ca0cf1660d83cfee6694376be2ed17b02ece8691f905b9b6a097dfed" Jul 22 17:02:03.075821 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.075777 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv"] Jul 22 17:02:03.076366 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.075992 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" podUID="87873f81-233d-45fb-8d28-b00ce410669c" containerName="maas-api" containerID="cri-o://2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59" gracePeriod=30 Jul 22 17:02:03.340601 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.340532 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:02:03.358139 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.358105 2580 generic.go:356] "Generic (PLEG): container finished" podID="87873f81-233d-45fb-8d28-b00ce410669c" containerID="2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59" exitCode=0 Jul 22 17:02:03.358310 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.358151 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" Jul 22 17:02:03.358310 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.358187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" event={"ID":"87873f81-233d-45fb-8d28-b00ce410669c","Type":"ContainerDied","Data":"2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59"} Jul 22 17:02:03.358310 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.358245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv" event={"ID":"87873f81-233d-45fb-8d28-b00ce410669c","Type":"ContainerDied","Data":"cc0b5d367ab0fe36d8c01f1a36c8cccb138f19572b91963d2b13c96512881ab3"} Jul 22 17:02:03.358310 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.358268 2580 scope.go:117] "RemoveContainer" containerID="2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59" Jul 22 17:02:03.370983 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.370952 2580 scope.go:117] "RemoveContainer" containerID="2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59" Jul 22 17:02:03.371430 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:02:03.371404 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59\": container with ID starting with 2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59 not found: ID does not exist" containerID="2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59" Jul 22 17:02:03.371505 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.371444 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59"} err="failed to get container status \"2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59\": rpc error: code = NotFound desc = could not find container \"2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59\": container with ID starting with 2d9841d16f59d217346e92e852021a537d3d8e8aaa2c214e9a1339dd90a55c59 not found: ID does not exist" Jul 22 17:02:03.449978 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.449941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87873f81-233d-45fb-8d28-b00ce410669c-openshift-service-ca\") pod \"87873f81-233d-45fb-8d28-b00ce410669c\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " Jul 22 17:02:03.450190 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.449993 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf5qn\" (UniqueName: \"kubernetes.io/projected/87873f81-233d-45fb-8d28-b00ce410669c-kube-api-access-pf5qn\") pod \"87873f81-233d-45fb-8d28-b00ce410669c\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " Jul 22 17:02:03.450190 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.450120 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87873f81-233d-45fb-8d28-b00ce410669c-maas-api-tls\") pod \"87873f81-233d-45fb-8d28-b00ce410669c\" (UID: \"87873f81-233d-45fb-8d28-b00ce410669c\") " Jul 22 17:02:03.450452 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.450423 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87873f81-233d-45fb-8d28-b00ce410669c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87873f81-233d-45fb-8d28-b00ce410669c" (UID: "87873f81-233d-45fb-8d28-b00ce410669c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:02:03.452235 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.452210 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87873f81-233d-45fb-8d28-b00ce410669c-kube-api-access-pf5qn" (OuterVolumeSpecName: "kube-api-access-pf5qn") pod "87873f81-233d-45fb-8d28-b00ce410669c" (UID: "87873f81-233d-45fb-8d28-b00ce410669c"). InnerVolumeSpecName "kube-api-access-pf5qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:02:03.452334 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.452212 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87873f81-233d-45fb-8d28-b00ce410669c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87873f81-233d-45fb-8d28-b00ce410669c" (UID: "87873f81-233d-45fb-8d28-b00ce410669c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:02:03.551490 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.551449 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87873f81-233d-45fb-8d28-b00ce410669c-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:02:03.551490 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.551485 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87873f81-233d-45fb-8d28-b00ce410669c-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:02:03.551490 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.551496 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf5qn\" (UniqueName: \"kubernetes.io/projected/87873f81-233d-45fb-8d28-b00ce410669c-kube-api-access-pf5qn\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:02:03.682138 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.682097 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv"] Jul 22 17:02:03.687104 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:03.687071 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf7663-c87bbd8f-6nxhv"] Jul 22 17:02:04.980195 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:04.980158 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87873f81-233d-45fb-8d28-b00ce410669c" path="/var/lib/kubelet/pods/87873f81-233d-45fb-8d28-b00ce410669c/volumes" Jul 22 17:02:30.161881 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.161844 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds"] Jul 22 17:02:30.162765 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.162726 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" podUID="63678b91-7d74-4315-80f3-1d14245b89a9" containerName="maas-api" containerID="cri-o://ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6" gracePeriod=30 Jul 22 17:02:30.430182 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.430150 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:02:30.434642 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.434614 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63678b91-7d74-4315-80f3-1d14245b89a9-maas-api-tls\") pod \"63678b91-7d74-4315-80f3-1d14245b89a9\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " Jul 22 17:02:30.434862 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.434699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzxrh\" (UniqueName: \"kubernetes.io/projected/63678b91-7d74-4315-80f3-1d14245b89a9-kube-api-access-mzxrh\") pod \"63678b91-7d74-4315-80f3-1d14245b89a9\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " Jul 22 17:02:30.434862 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.434801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63678b91-7d74-4315-80f3-1d14245b89a9-openshift-service-ca\") pod \"63678b91-7d74-4315-80f3-1d14245b89a9\" (UID: \"63678b91-7d74-4315-80f3-1d14245b89a9\") " Jul 22 17:02:30.435369 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.435335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63678b91-7d74-4315-80f3-1d14245b89a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63678b91-7d74-4315-80f3-1d14245b89a9" (UID: "63678b91-7d74-4315-80f3-1d14245b89a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:02:30.437229 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.437199 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63678b91-7d74-4315-80f3-1d14245b89a9-kube-api-access-mzxrh" (OuterVolumeSpecName: "kube-api-access-mzxrh") pod "63678b91-7d74-4315-80f3-1d14245b89a9" (UID: "63678b91-7d74-4315-80f3-1d14245b89a9"). InnerVolumeSpecName "kube-api-access-mzxrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:02:30.437431 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.437379 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63678b91-7d74-4315-80f3-1d14245b89a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63678b91-7d74-4315-80f3-1d14245b89a9" (UID: "63678b91-7d74-4315-80f3-1d14245b89a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:02:30.532532 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.532491 2580 generic.go:356] "Generic (PLEG): container finished" podID="63678b91-7d74-4315-80f3-1d14245b89a9" containerID="ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6" exitCode=0 Jul 22 17:02:30.532732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.532545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" Jul 22 17:02:30.532732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.532571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" event={"ID":"63678b91-7d74-4315-80f3-1d14245b89a9","Type":"ContainerDied","Data":"ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6"} Jul 22 17:02:30.532732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.532650 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds" event={"ID":"63678b91-7d74-4315-80f3-1d14245b89a9","Type":"ContainerDied","Data":"e0be4b8c699819c33832a99d528999e8dfcffabd493b81dddbeb71ce12ee2fa0"} Jul 22 17:02:30.532732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.532680 2580 scope.go:117] "RemoveContainer" containerID="ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6" Jul 22 17:02:30.536059 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.536032 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxrh\" (UniqueName: \"kubernetes.io/projected/63678b91-7d74-4315-80f3-1d14245b89a9-kube-api-access-mzxrh\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:02:30.536059 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.536062 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63678b91-7d74-4315-80f3-1d14245b89a9-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:02:30.536281 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.536077 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63678b91-7d74-4315-80f3-1d14245b89a9-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:02:30.544853 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.544829 2580 scope.go:117] "RemoveContainer" containerID="ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6" Jul 22 17:02:30.545150 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:02:30.545128 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6\": container with ID starting with ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6 not found: ID does not exist" containerID="ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6" Jul 22 17:02:30.545214 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.545158 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6"} err="failed to get container status \"ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6\": rpc error: code = NotFound desc = could not find container \"ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6\": container with ID starting with ea9fee58357315210a8e8fe33a5e3f623bfdd7936364e9d34d14ad25c29648b6 not found: ID does not exist" Jul 22 17:02:30.557458 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.557427 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds"] Jul 22 17:02:30.559968 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.559945 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f66aa-56c88dc544-bjcds"] Jul 22 17:02:30.980140 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:30.980103 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63678b91-7d74-4315-80f3-1d14245b89a9" path="/var/lib/kubelet/pods/63678b91-7d74-4315-80f3-1d14245b89a9/volumes" Jul 22 17:02:45.701409 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701356 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z"] Jul 22 17:02:45.701829 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701815 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87873f81-233d-45fb-8d28-b00ce410669c" containerName="maas-api" Jul 22 17:02:45.701876 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701830 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87873f81-233d-45fb-8d28-b00ce410669c" containerName="maas-api" Jul 22 17:02:45.701876 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701852 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63678b91-7d74-4315-80f3-1d14245b89a9" containerName="maas-api" Jul 22 17:02:45.701876 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701857 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63678b91-7d74-4315-80f3-1d14245b89a9" containerName="maas-api" Jul 22 17:02:45.701974 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701922 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87873f81-233d-45fb-8d28-b00ce410669c" containerName="maas-api" Jul 22 17:02:45.701974 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.701932 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63678b91-7d74-4315-80f3-1d14245b89a9" containerName="maas-api" Jul 22 17:02:45.710345 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.710314 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.713298 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.713252 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d6d924-serving-cert\"" Jul 22 17:02:45.715225 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.715204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z"] Jul 22 17:02:45.786317 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.786285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaaf8c-d7be-432b-afc1-a53e221bb98b-maas-api-tls\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.786514 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.786334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaaf8c-d7be-432b-afc1-a53e221bb98b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.786514 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.786458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94s9j\" (UniqueName: \"kubernetes.io/projected/fefaaf8c-d7be-432b-afc1-a53e221bb98b-kube-api-access-94s9j\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.887862 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.887828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94s9j\" (UniqueName: \"kubernetes.io/projected/fefaaf8c-d7be-432b-afc1-a53e221bb98b-kube-api-access-94s9j\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.888049 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.887914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaaf8c-d7be-432b-afc1-a53e221bb98b-maas-api-tls\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.888049 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.887955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaaf8c-d7be-432b-afc1-a53e221bb98b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.888769 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.888737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaaf8c-d7be-432b-afc1-a53e221bb98b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.890564 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.890544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaaf8c-d7be-432b-afc1-a53e221bb98b-maas-api-tls\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:45.897351 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:45.897329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94s9j\" (UniqueName: \"kubernetes.io/projected/fefaaf8c-d7be-432b-afc1-a53e221bb98b-kube-api-access-94s9j\") pod \"maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:46.023104 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:46.023067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:46.164027 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:46.163995 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z"] Jul 22 17:02:46.165254 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:02:46.165229 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefaaf8c_d7be_432b_afc1_a53e221bb98b.slice/crio-058beaef70807fce3d83bc4383bb886d766f1efab733d2f8e213d73927b4e47c WatchSource:0}: Error finding container 058beaef70807fce3d83bc4383bb886d766f1efab733d2f8e213d73927b4e47c: Status 404 returned error can't find the container with id 058beaef70807fce3d83bc4383bb886d766f1efab733d2f8e213d73927b4e47c Jul 22 17:02:46.632139 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:46.632106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" event={"ID":"fefaaf8c-d7be-432b-afc1-a53e221bb98b","Type":"ContainerStarted","Data":"058beaef70807fce3d83bc4383bb886d766f1efab733d2f8e213d73927b4e47c"} Jul 22 17:02:47.640922 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:47.640887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" event={"ID":"fefaaf8c-d7be-432b-afc1-a53e221bb98b","Type":"ContainerStarted","Data":"f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9"} Jul 22 17:02:47.641428 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:47.641009 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:47.660230 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:47.660179 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" podStartSLOduration=2.16644177 podStartE2EDuration="2.660161121s" totalImagesPullingTime="493.719351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:02:47.658416713 +0000 UTC m=+2525.238510639" watchObservedRunningTime="2026-07-22 17:02:47.660161121 +0000 UTC m=+2525.240255017" Jul 22 17:02:53.652173 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:53.652141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:02:53.889869 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:53.889836 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm"] Jul 22 17:02:53.893989 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:53.893953 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:53.896258 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:53.896236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-cfb31b-serving-cert\"" Jul 22 17:02:53.904704 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:53.904679 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm"] Jul 22 17:02:54.069541 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.069510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmdbw\" (UniqueName: \"kubernetes.io/projected/13543bf7-aa28-4608-96d2-406dc7fa8d28-kube-api-access-cmdbw\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.069732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.069557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13543bf7-aa28-4608-96d2-406dc7fa8d28-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.069732 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.069648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.170742 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.170645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmdbw\" (UniqueName: \"kubernetes.io/projected/13543bf7-aa28-4608-96d2-406dc7fa8d28-kube-api-access-cmdbw\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.170742 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.170699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13543bf7-aa28-4608-96d2-406dc7fa8d28-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.170742 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.170740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.171035 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:02:54.170828 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-serving-cert: secret "maas-api-e2e-shared-b-cfb31b-serving-cert" not found Jul 22 17:02:54.171035 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:02:54.170881 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls podName:13543bf7-aa28-4608-96d2-406dc7fa8d28 nodeName:}" failed. No retries permitted until 2026-07-22 17:02:54.67086568 +0000 UTC m=+2532.250959555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls") pod "maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" (UID: "13543bf7-aa28-4608-96d2-406dc7fa8d28") : secret "maas-api-e2e-shared-b-cfb31b-serving-cert" not found Jul 22 17:02:54.171448 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.171427 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13543bf7-aa28-4608-96d2-406dc7fa8d28-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.180487 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.180463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmdbw\" (UniqueName: \"kubernetes.io/projected/13543bf7-aa28-4608-96d2-406dc7fa8d28-kube-api-access-cmdbw\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.675706 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.675669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.678565 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.678539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls\") pod \"maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.808189 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.808132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:54.972376 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:54.972339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm"] Jul 22 17:02:54.975279 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:02:54.975255 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13543bf7_aa28_4608_96d2_406dc7fa8d28.slice/crio-db3847ad77a5064dabf6d6aef571b35be9080efaf3ebee29d2c6b70f4d1d04f0 WatchSource:0}: Error finding container db3847ad77a5064dabf6d6aef571b35be9080efaf3ebee29d2c6b70f4d1d04f0: Status 404 returned error can't find the container with id db3847ad77a5064dabf6d6aef571b35be9080efaf3ebee29d2c6b70f4d1d04f0 Jul 22 17:02:55.690503 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:55.690369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" event={"ID":"13543bf7-aa28-4608-96d2-406dc7fa8d28","Type":"ContainerStarted","Data":"6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421"} Jul 22 17:02:55.690503 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:55.690456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" event={"ID":"13543bf7-aa28-4608-96d2-406dc7fa8d28","Type":"ContainerStarted","Data":"db3847ad77a5064dabf6d6aef571b35be9080efaf3ebee29d2c6b70f4d1d04f0"} Jul 22 17:02:55.690503 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:55.690483 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:02:55.710192 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:02:55.710138 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" podStartSLOduration=2.333153325 podStartE2EDuration="2.710122887s" totalImagesPullingTime="376.969562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:02:55.707322433 +0000 UTC m=+2533.287416348" watchObservedRunningTime="2026-07-22 17:02:55.710122887 +0000 UTC m=+2533.290216840" Jul 22 17:03:01.702725 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:03:01.702695 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:06:43.401571 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:06:43.401536 2580 scope.go:117] "RemoveContainer" containerID="ca0995dfac7cae8038a1a4c09fabb57e0959594862d440cc4fe0be6f502e2912" Jul 22 17:07:43.424243 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:07:43.424153 2580 scope.go:117] "RemoveContainer" containerID="3552588634d292db26bf3c9f5971cf78bcf2cb8a2ccd094132433803a295f8b4" Jul 22 17:09:37.931898 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:37.931863 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7142-140328263759280"] Jul 22 17:09:37.935308 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:37.935286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:37.937512 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:37.937488 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s4kvd\"" Jul 22 17:09:37.944260 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:37.944236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263759280"] Jul 22 17:09:38.017465 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:38.017436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgtf\" (UniqueName: \"kubernetes.io/projected/17688ead-b39d-4a68-b75c-81dc8d498e82-kube-api-access-wwgtf\") pod \"test-curl-7142-140328263759280\" (UID: \"17688ead-b39d-4a68-b75c-81dc8d498e82\") " pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:38.118582 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:38.118534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgtf\" (UniqueName: \"kubernetes.io/projected/17688ead-b39d-4a68-b75c-81dc8d498e82-kube-api-access-wwgtf\") pod \"test-curl-7142-140328263759280\" (UID: \"17688ead-b39d-4a68-b75c-81dc8d498e82\") " pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:38.126986 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:38.126959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgtf\" (UniqueName: \"kubernetes.io/projected/17688ead-b39d-4a68-b75c-81dc8d498e82-kube-api-access-wwgtf\") pod \"test-curl-7142-140328263759280\" (UID: \"17688ead-b39d-4a68-b75c-81dc8d498e82\") " pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:38.246180 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:38.246136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:38.383799 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:38.383769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263759280"] Jul 22 17:09:38.384622 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:09:38.384598 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17688ead_b39d_4a68_b75c_81dc8d498e82.slice/crio-a7631a20dd25c75e8e06a49b4554579126a80fb2297b7d487eeb859e274fd815 WatchSource:0}: Error finding container a7631a20dd25c75e8e06a49b4554579126a80fb2297b7d487eeb859e274fd815: Status 404 returned error can't find the container with id a7631a20dd25c75e8e06a49b4554579126a80fb2297b7d487eeb859e274fd815 Jul 22 17:09:38.386054 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:38.386030 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:09:39.146384 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:39.146347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263759280" event={"ID":"17688ead-b39d-4a68-b75c-81dc8d498e82","Type":"ContainerStarted","Data":"6999f0dfb4b4b7c56c55c7ac247760d788f857d5ac8df771afc4281aed635426"} Jul 22 17:09:39.146754 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:39.146419 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263759280" event={"ID":"17688ead-b39d-4a68-b75c-81dc8d498e82","Type":"ContainerStarted","Data":"a7631a20dd25c75e8e06a49b4554579126a80fb2297b7d487eeb859e274fd815"} Jul 22 17:09:39.162426 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:39.162356 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7142-140328263759280" podStartSLOduration=1.470976854 podStartE2EDuration="2.162337468s" totalImagesPullingTime="691.360614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:09:39.159956 +0000 UTC m=+2936.740049917" watchObservedRunningTime="2026-07-22 17:09:39.162337468 +0000 UTC m=+2936.742431365" Jul 22 17:09:39.261908 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:39.261875 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7142-140328263759280_17688ead-b39d-4a68-b75c-81dc8d498e82/test-curl-7142-140328263759280/0.log" Jul 22 17:09:40.154086 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:40.154052 2580 generic.go:356] "Generic (PLEG): container finished" podID="17688ead-b39d-4a68-b75c-81dc8d498e82" containerID="6999f0dfb4b4b7c56c55c7ac247760d788f857d5ac8df771afc4281aed635426" exitCode=0 Jul 22 17:09:40.154471 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:40.154144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263759280" event={"ID":"17688ead-b39d-4a68-b75c-81dc8d498e82","Type":"ContainerDied","Data":"6999f0dfb4b4b7c56c55c7ac247760d788f857d5ac8df771afc4281aed635426"} Jul 22 17:09:41.303543 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.303522 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:41.343733 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.343696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7142-140328263759280"] Jul 22 17:09:41.347415 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.347355 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7142-140328263759280"] Jul 22 17:09:41.348199 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.348176 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwgtf\" (UniqueName: \"kubernetes.io/projected/17688ead-b39d-4a68-b75c-81dc8d498e82-kube-api-access-wwgtf\") pod \"17688ead-b39d-4a68-b75c-81dc8d498e82\" (UID: \"17688ead-b39d-4a68-b75c-81dc8d498e82\") " Jul 22 17:09:41.350267 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.350241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17688ead-b39d-4a68-b75c-81dc8d498e82-kube-api-access-wwgtf" (OuterVolumeSpecName: "kube-api-access-wwgtf") pod "17688ead-b39d-4a68-b75c-81dc8d498e82" (UID: "17688ead-b39d-4a68-b75c-81dc8d498e82"). InnerVolumeSpecName "kube-api-access-wwgtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:41.449687 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.449592 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgtf\" (UniqueName: \"kubernetes.io/projected/17688ead-b39d-4a68-b75c-81dc8d498e82-kube-api-access-wwgtf\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:09:41.496751 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.496716 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7142-140328263760304"] Jul 22 17:09:41.497246 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.497227 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17688ead-b39d-4a68-b75c-81dc8d498e82" containerName="test-curl-7142-140328263759280" Jul 22 17:09:41.497336 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.497249 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17688ead-b39d-4a68-b75c-81dc8d498e82" containerName="test-curl-7142-140328263759280" Jul 22 17:09:41.497410 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.497370 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17688ead-b39d-4a68-b75c-81dc8d498e82" containerName="test-curl-7142-140328263759280" Jul 22 17:09:41.500645 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.500624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:41.507088 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.507037 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263760304"] Jul 22 17:09:41.550164 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.550128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpqq\" (UniqueName: \"kubernetes.io/projected/1b218525-57aa-4f52-a748-810472195372-kube-api-access-vkpqq\") pod \"test-curl-7142-140328263760304\" (UID: \"1b218525-57aa-4f52-a748-810472195372\") " pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:41.651682 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.651637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpqq\" (UniqueName: \"kubernetes.io/projected/1b218525-57aa-4f52-a748-810472195372-kube-api-access-vkpqq\") pod \"test-curl-7142-140328263760304\" (UID: \"1b218525-57aa-4f52-a748-810472195372\") " pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:41.660493 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.660468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpqq\" (UniqueName: \"kubernetes.io/projected/1b218525-57aa-4f52-a748-810472195372-kube-api-access-vkpqq\") pod \"test-curl-7142-140328263760304\" (UID: \"1b218525-57aa-4f52-a748-810472195372\") " pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:41.812351 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.812317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:41.970964 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:41.970928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263760304"] Jul 22 17:09:41.973757 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:09:41.973720 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b218525_57aa_4f52_a748_810472195372.slice/crio-72421eca466fddc8bd485eeed06f86e2e234be9feda2ec4021855beff9b51f58 WatchSource:0}: Error finding container 72421eca466fddc8bd485eeed06f86e2e234be9feda2ec4021855beff9b51f58: Status 404 returned error can't find the container with id 72421eca466fddc8bd485eeed06f86e2e234be9feda2ec4021855beff9b51f58 Jul 22 17:09:42.167136 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:42.167048 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:42.167136 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:42.167063 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7631a20dd25c75e8e06a49b4554579126a80fb2297b7d487eeb859e274fd815" Jul 22 17:09:42.168247 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:42.168216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263760304" event={"ID":"1b218525-57aa-4f52-a748-810472195372","Type":"ContainerStarted","Data":"72421eca466fddc8bd485eeed06f86e2e234be9feda2ec4021855beff9b51f58"} Jul 22 17:09:42.177789 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:09:42.177762 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7142-140328263759280\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="17688ead-b39d-4a68-b75c-81dc8d498e82" pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:42.979809 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:09:42.979773 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7142-140328263759280\" is forbidden: User \"system:node:ip-10-0-133-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-136.ec2.internal' and this object" podUID="17688ead-b39d-4a68-b75c-81dc8d498e82" pod="opendatahub/test-curl-7142-140328263759280" Jul 22 17:09:42.981656 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:42.981635 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17688ead-b39d-4a68-b75c-81dc8d498e82" path="/var/lib/kubelet/pods/17688ead-b39d-4a68-b75c-81dc8d498e82/volumes" Jul 22 17:09:43.175089 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:43.175057 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b218525-57aa-4f52-a748-810472195372" containerID="882ab1bb306b5a90cd45f56278335666971135083d906157b230587d888b8ef2" exitCode=0 Jul 22 17:09:43.175277 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:43.175146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263760304" event={"ID":"1b218525-57aa-4f52-a748-810472195372","Type":"ContainerDied","Data":"882ab1bb306b5a90cd45f56278335666971135083d906157b230587d888b8ef2"} Jul 22 17:09:44.326949 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.326924 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:44.347232 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.347212 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7142-140328263760304_1b218525-57aa-4f52-a748-810472195372/test-curl-7142-140328263760304/0.log" Jul 22 17:09:44.372753 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.372723 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7142-140328263760304"] Jul 22 17:09:44.377326 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.377306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkpqq\" (UniqueName: \"kubernetes.io/projected/1b218525-57aa-4f52-a748-810472195372-kube-api-access-vkpqq\") pod \"1b218525-57aa-4f52-a748-810472195372\" (UID: \"1b218525-57aa-4f52-a748-810472195372\") " Jul 22 17:09:44.378620 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.378585 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7142-140328263760304"] Jul 22 17:09:44.379336 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.379308 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b218525-57aa-4f52-a748-810472195372-kube-api-access-vkpqq" (OuterVolumeSpecName: "kube-api-access-vkpqq") pod "1b218525-57aa-4f52-a748-810472195372" (UID: "1b218525-57aa-4f52-a748-810472195372"). InnerVolumeSpecName "kube-api-access-vkpqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:44.478748 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.478673 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpqq\" (UniqueName: \"kubernetes.io/projected/1b218525-57aa-4f52-a748-810472195372-kube-api-access-vkpqq\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:09:44.546131 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.546100 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:44.546597 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.546583 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b218525-57aa-4f52-a748-810472195372" containerName="test-curl-7142-140328263760304" Jul 22 17:09:44.546648 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.546599 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b218525-57aa-4f52-a748-810472195372" containerName="test-curl-7142-140328263760304" Jul 22 17:09:44.546693 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.546683 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b218525-57aa-4f52-a748-810472195372" containerName="test-curl-7142-140328263760304" Jul 22 17:09:44.549815 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.549797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:44.554416 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.554373 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:44.681151 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.681111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8gb\" (UniqueName: \"kubernetes.io/projected/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca-kube-api-access-rf8gb\") pod \"test-curl-7142-140328263717920\" (UID: \"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca\") " pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:44.782041 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.781986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8gb\" (UniqueName: \"kubernetes.io/projected/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca-kube-api-access-rf8gb\") pod \"test-curl-7142-140328263717920\" (UID: \"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca\") " pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:44.791522 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.791499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8gb\" (UniqueName: \"kubernetes.io/projected/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca-kube-api-access-rf8gb\") pod \"test-curl-7142-140328263717920\" (UID: \"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca\") " pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:44.862931 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.862880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:44.982933 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.982894 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b218525-57aa-4f52-a748-810472195372" path="/var/lib/kubelet/pods/1b218525-57aa-4f52-a748-810472195372/volumes" Jul 22 17:09:44.997072 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:44.997046 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:44.998633 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:09:44.998610 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7075dd45_88c9_4bbc_afc0_1f06ea3f5eca.slice/crio-241600ab11de9d561a1443f871010f069c72454efe28ee1a060efa0ceaa6a402 WatchSource:0}: Error finding container 241600ab11de9d561a1443f871010f069c72454efe28ee1a060efa0ceaa6a402: Status 404 returned error can't find the container with id 241600ab11de9d561a1443f871010f069c72454efe28ee1a060efa0ceaa6a402 Jul 22 17:09:45.196623 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:45.196586 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263760304" Jul 22 17:09:45.196623 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:45.196612 2580 scope.go:117] "RemoveContainer" containerID="882ab1bb306b5a90cd45f56278335666971135083d906157b230587d888b8ef2" Jul 22 17:09:45.197968 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:45.197741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263717920" event={"ID":"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca","Type":"ContainerStarted","Data":"241600ab11de9d561a1443f871010f069c72454efe28ee1a060efa0ceaa6a402"} Jul 22 17:09:46.205420 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:46.205360 2580 generic.go:356] "Generic (PLEG): container finished" podID="7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" containerID="862cdbb26fd004fcf935fccdc18bdb0ea6cfd7019261707afdde6776b0124725" exitCode=0 Jul 22 17:09:46.205792 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:46.205423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263717920" event={"ID":"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca","Type":"ContainerDied","Data":"862cdbb26fd004fcf935fccdc18bdb0ea6cfd7019261707afdde6776b0124725"} Jul 22 17:09:47.352553 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.352526 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:47.373215 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.373164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7142-140328263717920_7075dd45-88c9-4bbc-afc0-1f06ea3f5eca/test-curl-7142-140328263717920/0.log" Jul 22 17:09:47.399760 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.399732 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:47.404190 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.404167 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:47.415691 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.415671 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf8gb\" (UniqueName: \"kubernetes.io/projected/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca-kube-api-access-rf8gb\") pod \"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca\" (UID: \"7075dd45-88c9-4bbc-afc0-1f06ea3f5eca\") " Jul 22 17:09:47.417734 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.417711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca-kube-api-access-rf8gb" (OuterVolumeSpecName: "kube-api-access-rf8gb") pod "7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" (UID: "7075dd45-88c9-4bbc-afc0-1f06ea3f5eca"). InnerVolumeSpecName "kube-api-access-rf8gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:47.516833 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.516803 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf8gb\" (UniqueName: \"kubernetes.io/projected/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca-kube-api-access-rf8gb\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:09:47.538056 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.538019 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:47.538723 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.538703 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" containerName="test-curl-7142-140328263717920" Jul 22 17:09:47.538820 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.538725 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" containerName="test-curl-7142-140328263717920" Jul 22 17:09:47.538871 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.538860 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" containerName="test-curl-7142-140328263717920" Jul 22 17:09:47.542524 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.542501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:47.545655 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.545631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:47.617506 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.617461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntrv\" (UniqueName: \"kubernetes.io/projected/90073cc8-7270-4b75-a6c0-7be002d2e962-kube-api-access-sntrv\") pod \"test-curl-7142-140328263717920\" (UID: \"90073cc8-7270-4b75-a6c0-7be002d2e962\") " pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:47.718164 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.718126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntrv\" (UniqueName: \"kubernetes.io/projected/90073cc8-7270-4b75-a6c0-7be002d2e962-kube-api-access-sntrv\") pod \"test-curl-7142-140328263717920\" (UID: \"90073cc8-7270-4b75-a6c0-7be002d2e962\") " pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:47.726755 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.726729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntrv\" (UniqueName: \"kubernetes.io/projected/90073cc8-7270-4b75-a6c0-7be002d2e962-kube-api-access-sntrv\") pod \"test-curl-7142-140328263717920\" (UID: \"90073cc8-7270-4b75-a6c0-7be002d2e962\") " pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:47.854311 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.854232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:47.999137 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:47.999107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:48.000253 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:09:48.000217 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90073cc8_7270_4b75_a6c0_7be002d2e962.slice/crio-4dc3f8e4463473c0bbfa2f658da7ad2dcd596a618b7329a28f168b15607ac501 WatchSource:0}: Error finding container 4dc3f8e4463473c0bbfa2f658da7ad2dcd596a618b7329a28f168b15607ac501: Status 404 returned error can't find the container with id 4dc3f8e4463473c0bbfa2f658da7ad2dcd596a618b7329a28f168b15607ac501 Jul 22 17:09:48.218950 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:48.218862 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:48.218950 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:48.218911 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="241600ab11de9d561a1443f871010f069c72454efe28ee1a060efa0ceaa6a402" Jul 22 17:09:48.220248 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:48.220222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263717920" event={"ID":"90073cc8-7270-4b75-a6c0-7be002d2e962","Type":"ContainerStarted","Data":"4dc3f8e4463473c0bbfa2f658da7ad2dcd596a618b7329a28f168b15607ac501"} Jul 22 17:09:48.231255 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:48.231223 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7142-140328263717920" oldPodUID="7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" podUID="90073cc8-7270-4b75-a6c0-7be002d2e962" Jul 22 17:09:48.981701 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:48.981664 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7075dd45-88c9-4bbc-afc0-1f06ea3f5eca" path="/var/lib/kubelet/pods/7075dd45-88c9-4bbc-afc0-1f06ea3f5eca/volumes" Jul 22 17:09:49.228440 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:49.228386 2580 generic.go:356] "Generic (PLEG): container finished" podID="90073cc8-7270-4b75-a6c0-7be002d2e962" containerID="e922bd1be44fb2516a4f7ececf87b9674c532728c89f9baffd25a9b3beda256a" exitCode=0 Jul 22 17:09:49.228593 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:49.228488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263717920" event={"ID":"90073cc8-7270-4b75-a6c0-7be002d2e962","Type":"ContainerDied","Data":"e922bd1be44fb2516a4f7ececf87b9674c532728c89f9baffd25a9b3beda256a"} Jul 22 17:09:50.381888 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.381868 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:50.402457 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.402431 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7142-140328263717920_90073cc8-7270-4b75-a6c0-7be002d2e962/test-curl-7142-140328263717920/0.log" Jul 22 17:09:50.430244 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.430209 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:50.441787 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.441749 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7142-140328263717920"] Jul 22 17:09:50.445540 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.445516 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sntrv\" (UniqueName: \"kubernetes.io/projected/90073cc8-7270-4b75-a6c0-7be002d2e962-kube-api-access-sntrv\") pod \"90073cc8-7270-4b75-a6c0-7be002d2e962\" (UID: \"90073cc8-7270-4b75-a6c0-7be002d2e962\") " Jul 22 17:09:50.447565 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.447539 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90073cc8-7270-4b75-a6c0-7be002d2e962-kube-api-access-sntrv" (OuterVolumeSpecName: "kube-api-access-sntrv") pod "90073cc8-7270-4b75-a6c0-7be002d2e962" (UID: "90073cc8-7270-4b75-a6c0-7be002d2e962"). InnerVolumeSpecName "kube-api-access-sntrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:50.546345 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.546311 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sntrv\" (UniqueName: \"kubernetes.io/projected/90073cc8-7270-4b75-a6c0-7be002d2e962-kube-api-access-sntrv\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:09:50.575215 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.575182 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:50.575692 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.575677 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90073cc8-7270-4b75-a6c0-7be002d2e962" containerName="test-curl-7142-140328263717920" Jul 22 17:09:50.575745 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.575693 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="90073cc8-7270-4b75-a6c0-7be002d2e962" containerName="test-curl-7142-140328263717920" Jul 22 17:09:50.575800 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.575791 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="90073cc8-7270-4b75-a6c0-7be002d2e962" containerName="test-curl-7142-140328263717920" Jul 22 17:09:50.578945 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.578929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:50.595068 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.595047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:50.647552 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.647522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjcs\" (UniqueName: \"kubernetes.io/projected/f0e2d440-3635-4235-be43-7014ef31528d-kube-api-access-9mjcs\") pod \"test-curl-7142-140328263718928\" (UID: \"f0e2d440-3635-4235-be43-7014ef31528d\") " pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:50.748907 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.748870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjcs\" (UniqueName: \"kubernetes.io/projected/f0e2d440-3635-4235-be43-7014ef31528d-kube-api-access-9mjcs\") pod \"test-curl-7142-140328263718928\" (UID: \"f0e2d440-3635-4235-be43-7014ef31528d\") " pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:50.758425 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.758387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjcs\" (UniqueName: \"kubernetes.io/projected/f0e2d440-3635-4235-be43-7014ef31528d-kube-api-access-9mjcs\") pod \"test-curl-7142-140328263718928\" (UID: \"f0e2d440-3635-4235-be43-7014ef31528d\") " pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:50.888000 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.887918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:50.983190 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:50.983156 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90073cc8-7270-4b75-a6c0-7be002d2e962" path="/var/lib/kubelet/pods/90073cc8-7270-4b75-a6c0-7be002d2e962/volumes" Jul 22 17:09:51.028248 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:51.028224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:51.029775 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:09:51.029750 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0e2d440_3635_4235_be43_7014ef31528d.slice/crio-e6274df87774c6b8f1a8ab0000fe90673980d65d7fc46275a7bb5ece7d9fac39 WatchSource:0}: Error finding container e6274df87774c6b8f1a8ab0000fe90673980d65d7fc46275a7bb5ece7d9fac39: Status 404 returned error can't find the container with id e6274df87774c6b8f1a8ab0000fe90673980d65d7fc46275a7bb5ece7d9fac39 Jul 22 17:09:51.240771 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:51.240745 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263717920" Jul 22 17:09:51.240771 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:51.240776 2580 scope.go:117] "RemoveContainer" containerID="e922bd1be44fb2516a4f7ececf87b9674c532728c89f9baffd25a9b3beda256a" Jul 22 17:09:51.242264 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:51.242233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263718928" event={"ID":"f0e2d440-3635-4235-be43-7014ef31528d","Type":"ContainerStarted","Data":"e6274df87774c6b8f1a8ab0000fe90673980d65d7fc46275a7bb5ece7d9fac39"} Jul 22 17:09:52.249221 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:52.249180 2580 generic.go:356] "Generic (PLEG): container finished" podID="f0e2d440-3635-4235-be43-7014ef31528d" containerID="fd61db6bff7d6302935511fce50803cec19a8e5e6cb1cb57c7d336e5a959bba9" exitCode=0 Jul 22 17:09:52.249695 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:52.249249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263718928" event={"ID":"f0e2d440-3635-4235-be43-7014ef31528d","Type":"ContainerDied","Data":"fd61db6bff7d6302935511fce50803cec19a8e5e6cb1cb57c7d336e5a959bba9"} Jul 22 17:09:53.403000 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.402975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:53.423230 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.423202 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7142-140328263718928_f0e2d440-3635-4235-be43-7014ef31528d/test-curl-7142-140328263718928/0.log" Jul 22 17:09:53.446638 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.446601 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:53.449113 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.449088 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:53.475323 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.475286 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mjcs\" (UniqueName: \"kubernetes.io/projected/f0e2d440-3635-4235-be43-7014ef31528d-kube-api-access-9mjcs\") pod \"f0e2d440-3635-4235-be43-7014ef31528d\" (UID: \"f0e2d440-3635-4235-be43-7014ef31528d\") " Jul 22 17:09:53.477458 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.477425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e2d440-3635-4235-be43-7014ef31528d-kube-api-access-9mjcs" (OuterVolumeSpecName: "kube-api-access-9mjcs") pod "f0e2d440-3635-4235-be43-7014ef31528d" (UID: "f0e2d440-3635-4235-be43-7014ef31528d"). InnerVolumeSpecName "kube-api-access-9mjcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:53.576907 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.576819 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjcs\" (UniqueName: \"kubernetes.io/projected/f0e2d440-3635-4235-be43-7014ef31528d-kube-api-access-9mjcs\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:09:53.582354 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.582328 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:53.582823 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.582808 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e2d440-3635-4235-be43-7014ef31528d" containerName="test-curl-7142-140328263718928" Jul 22 17:09:53.582880 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.582826 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e2d440-3635-4235-be43-7014ef31528d" containerName="test-curl-7142-140328263718928" Jul 22 17:09:53.582915 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.582884 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e2d440-3635-4235-be43-7014ef31528d" containerName="test-curl-7142-140328263718928" Jul 22 17:09:53.586005 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.585989 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:53.602413 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.595528 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:53.678134 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.678094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhjc\" (UniqueName: \"kubernetes.io/projected/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6-kube-api-access-gwhjc\") pod \"test-curl-7142-140328263718928\" (UID: \"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6\") " pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:53.779596 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.779553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhjc\" (UniqueName: \"kubernetes.io/projected/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6-kube-api-access-gwhjc\") pod \"test-curl-7142-140328263718928\" (UID: \"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6\") " pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:53.788382 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.788347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhjc\" (UniqueName: \"kubernetes.io/projected/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6-kube-api-access-gwhjc\") pod \"test-curl-7142-140328263718928\" (UID: \"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6\") " pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:53.907936 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:53.907825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:54.047233 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:54.047204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:54.049281 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:09:54.049255 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fdf01a3_cabe_47ad_aa80_2f1a483c30b6.slice/crio-ae7bf3291efa9f5d18dd1abeb5b2ac65f931fd127e53cab8f8cb250c630c25fa WatchSource:0}: Error finding container ae7bf3291efa9f5d18dd1abeb5b2ac65f931fd127e53cab8f8cb250c630c25fa: Status 404 returned error can't find the container with id ae7bf3291efa9f5d18dd1abeb5b2ac65f931fd127e53cab8f8cb250c630c25fa Jul 22 17:09:54.263159 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:54.263122 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:54.263317 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:54.263165 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6274df87774c6b8f1a8ab0000fe90673980d65d7fc46275a7bb5ece7d9fac39" Jul 22 17:09:54.264459 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:54.264430 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263718928" event={"ID":"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6","Type":"ContainerStarted","Data":"ae7bf3291efa9f5d18dd1abeb5b2ac65f931fd127e53cab8f8cb250c630c25fa"} Jul 22 17:09:54.275440 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:54.275416 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7142-140328263718928" oldPodUID="f0e2d440-3635-4235-be43-7014ef31528d" podUID="5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" Jul 22 17:09:54.980643 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:54.980608 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e2d440-3635-4235-be43-7014ef31528d" path="/var/lib/kubelet/pods/f0e2d440-3635-4235-be43-7014ef31528d/volumes" Jul 22 17:09:55.271299 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:55.271261 2580 generic.go:356] "Generic (PLEG): container finished" podID="5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" containerID="45e876e7aa8710d09687ca6ba7f7eae9d2ae59fb3c55549ed56d4e2ea5c52496" exitCode=0 Jul 22 17:09:55.271501 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:55.271345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7142-140328263718928" event={"ID":"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6","Type":"ContainerDied","Data":"45e876e7aa8710d09687ca6ba7f7eae9d2ae59fb3c55549ed56d4e2ea5c52496"} Jul 22 17:09:56.431710 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.431687 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:56.452923 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.452902 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7142-140328263718928_5fdf01a3-cabe-47ad-aa80-2f1a483c30b6/test-curl-7142-140328263718928/0.log" Jul 22 17:09:56.476914 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.476889 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:56.479440 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.479418 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7142-140328263718928"] Jul 22 17:09:56.606744 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.606643 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwhjc\" (UniqueName: \"kubernetes.io/projected/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6-kube-api-access-gwhjc\") pod \"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6\" (UID: \"5fdf01a3-cabe-47ad-aa80-2f1a483c30b6\") " Jul 22 17:09:56.608709 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.608676 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6-kube-api-access-gwhjc" (OuterVolumeSpecName: "kube-api-access-gwhjc") pod "5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" (UID: "5fdf01a3-cabe-47ad-aa80-2f1a483c30b6"). InnerVolumeSpecName "kube-api-access-gwhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:56.708214 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.708183 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhjc\" (UniqueName: \"kubernetes.io/projected/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6-kube-api-access-gwhjc\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:09:56.981282 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:56.981197 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" path="/var/lib/kubelet/pods/5fdf01a3-cabe-47ad-aa80-2f1a483c30b6/volumes" Jul 22 17:09:57.284335 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:57.284300 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7142-140328263718928" Jul 22 17:09:57.284511 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:09:57.284361 2580 scope.go:117] "RemoveContainer" containerID="45e876e7aa8710d09687ca6ba7f7eae9d2ae59fb3c55549ed56d4e2ea5c52496" Jul 22 17:10:07.118935 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.118853 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z"] Jul 22 17:10:07.119468 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.119070 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" podUID="fefaaf8c-d7be-432b-afc1-a53e221bb98b" containerName="maas-api" containerID="cri-o://f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9" gracePeriod=30 Jul 22 17:10:07.582183 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.582158 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:10:07.608868 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.608839 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94s9j\" (UniqueName: \"kubernetes.io/projected/fefaaf8c-d7be-432b-afc1-a53e221bb98b-kube-api-access-94s9j\") pod \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " Jul 22 17:10:07.609044 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.608922 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaaf8c-d7be-432b-afc1-a53e221bb98b-openshift-service-ca\") pod \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " Jul 22 17:10:07.609044 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.608975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaaf8c-d7be-432b-afc1-a53e221bb98b-maas-api-tls\") pod \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\" (UID: \"fefaaf8c-d7be-432b-afc1-a53e221bb98b\") " Jul 22 17:10:07.609342 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.609311 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefaaf8c-d7be-432b-afc1-a53e221bb98b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fefaaf8c-d7be-432b-afc1-a53e221bb98b" (UID: "fefaaf8c-d7be-432b-afc1-a53e221bb98b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:07.611197 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.611177 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefaaf8c-d7be-432b-afc1-a53e221bb98b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fefaaf8c-d7be-432b-afc1-a53e221bb98b" (UID: "fefaaf8c-d7be-432b-afc1-a53e221bb98b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:10:07.611292 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.611210 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefaaf8c-d7be-432b-afc1-a53e221bb98b-kube-api-access-94s9j" (OuterVolumeSpecName: "kube-api-access-94s9j") pod "fefaaf8c-d7be-432b-afc1-a53e221bb98b" (UID: "fefaaf8c-d7be-432b-afc1-a53e221bb98b"). InnerVolumeSpecName "kube-api-access-94s9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:07.710104 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.710012 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94s9j\" (UniqueName: \"kubernetes.io/projected/fefaaf8c-d7be-432b-afc1-a53e221bb98b-kube-api-access-94s9j\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:10:07.710104 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.710044 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaaf8c-d7be-432b-afc1-a53e221bb98b-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:10:07.710104 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:07.710054 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaaf8c-d7be-432b-afc1-a53e221bb98b-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:10:08.352670 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.352631 2580 generic.go:356] "Generic (PLEG): container finished" podID="fefaaf8c-d7be-432b-afc1-a53e221bb98b" containerID="f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9" exitCode=0 Jul 22 17:10:08.353067 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.352687 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" Jul 22 17:10:08.353067 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.352718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" event={"ID":"fefaaf8c-d7be-432b-afc1-a53e221bb98b","Type":"ContainerDied","Data":"f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9"} Jul 22 17:10:08.353067 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.352780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z" event={"ID":"fefaaf8c-d7be-432b-afc1-a53e221bb98b","Type":"ContainerDied","Data":"058beaef70807fce3d83bc4383bb886d766f1efab733d2f8e213d73927b4e47c"} Jul 22 17:10:08.353067 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.352802 2580 scope.go:117] "RemoveContainer" containerID="f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9" Jul 22 17:10:08.362641 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.362622 2580 scope.go:117] "RemoveContainer" containerID="f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9" Jul 22 17:10:08.362937 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:10:08.362919 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9\": container with ID starting with f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9 not found: ID does not exist" containerID="f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9" Jul 22 17:10:08.363013 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.362945 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9"} err="failed to get container status \"f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9\": rpc error: code = NotFound desc = could not find container \"f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9\": container with ID starting with f56f56e6262ece669f80c58dccb4d454c28a53ac9c2c5e6e835b9868129bd5e9 not found: ID does not exist" Jul 22 17:10:08.376176 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.376147 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z"] Jul 22 17:10:08.380986 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.380961 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d6d924-65b48c48c-lv99z"] Jul 22 17:10:08.980538 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:08.980500 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefaaf8c-d7be-432b-afc1-a53e221bb98b" path="/var/lib/kubelet/pods/fefaaf8c-d7be-432b-afc1-a53e221bb98b/volumes" Jul 22 17:10:29.893785 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:29.893747 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm"] Jul 22 17:10:29.894242 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:29.894025 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" podUID="13543bf7-aa28-4608-96d2-406dc7fa8d28" containerName="maas-api" containerID="cri-o://6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421" gracePeriod=30 Jul 22 17:10:30.357839 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.357816 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:10:30.414954 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.414925 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13543bf7-aa28-4608-96d2-406dc7fa8d28-openshift-service-ca\") pod \"13543bf7-aa28-4608-96d2-406dc7fa8d28\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " Jul 22 17:10:30.415106 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.414967 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls\") pod \"13543bf7-aa28-4608-96d2-406dc7fa8d28\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " Jul 22 17:10:30.415106 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.415043 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmdbw\" (UniqueName: \"kubernetes.io/projected/13543bf7-aa28-4608-96d2-406dc7fa8d28-kube-api-access-cmdbw\") pod \"13543bf7-aa28-4608-96d2-406dc7fa8d28\" (UID: \"13543bf7-aa28-4608-96d2-406dc7fa8d28\") " Jul 22 17:10:30.415321 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.415294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13543bf7-aa28-4608-96d2-406dc7fa8d28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13543bf7-aa28-4608-96d2-406dc7fa8d28" (UID: "13543bf7-aa28-4608-96d2-406dc7fa8d28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:30.417000 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.416974 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13543bf7-aa28-4608-96d2-406dc7fa8d28" (UID: "13543bf7-aa28-4608-96d2-406dc7fa8d28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:10:30.417097 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.417029 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13543bf7-aa28-4608-96d2-406dc7fa8d28-kube-api-access-cmdbw" (OuterVolumeSpecName: "kube-api-access-cmdbw") pod "13543bf7-aa28-4608-96d2-406dc7fa8d28" (UID: "13543bf7-aa28-4608-96d2-406dc7fa8d28"). InnerVolumeSpecName "kube-api-access-cmdbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:30.486492 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.486457 2580 generic.go:356] "Generic (PLEG): container finished" podID="13543bf7-aa28-4608-96d2-406dc7fa8d28" containerID="6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421" exitCode=0 Jul 22 17:10:30.486651 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.486510 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" Jul 22 17:10:30.486651 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.486545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" event={"ID":"13543bf7-aa28-4608-96d2-406dc7fa8d28","Type":"ContainerDied","Data":"6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421"} Jul 22 17:10:30.486651 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.486597 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm" event={"ID":"13543bf7-aa28-4608-96d2-406dc7fa8d28","Type":"ContainerDied","Data":"db3847ad77a5064dabf6d6aef571b35be9080efaf3ebee29d2c6b70f4d1d04f0"} Jul 22 17:10:30.486651 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.486625 2580 scope.go:117] "RemoveContainer" containerID="6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421" Jul 22 17:10:30.499259 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.499241 2580 scope.go:117] "RemoveContainer" containerID="6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421" Jul 22 17:10:30.499561 ip-10-0-133-136 kubenswrapper[2580]: E0722 17:10:30.499543 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421\": container with ID starting with 6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421 not found: ID does not exist" containerID="6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421" Jul 22 17:10:30.499622 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.499569 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421"} err="failed to get container status \"6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421\": rpc error: code = NotFound desc = could not find container \"6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421\": container with ID starting with 6c2756d00effc2a7169af60bdf9a40767ce7d9678452edc45725f00725cf3421 not found: ID does not exist" Jul 22 17:10:30.512894 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.512868 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm"] Jul 22 17:10:30.515352 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.515334 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cfb31b-664d7b65c4-x5lwm"] Jul 22 17:10:30.515833 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.515813 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13543bf7-aa28-4608-96d2-406dc7fa8d28-openshift-service-ca\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:10:30.515903 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.515838 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13543bf7-aa28-4608-96d2-406dc7fa8d28-maas-api-tls\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:10:30.515903 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.515855 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmdbw\" (UniqueName: \"kubernetes.io/projected/13543bf7-aa28-4608-96d2-406dc7fa8d28-kube-api-access-cmdbw\") on node \"ip-10-0-133-136.ec2.internal\" DevicePath \"\"" Jul 22 17:10:30.980537 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:30.980501 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13543bf7-aa28-4608-96d2-406dc7fa8d28" path="/var/lib/kubelet/pods/13543bf7-aa28-4608-96d2-406dc7fa8d28/volumes" Jul 22 17:10:42.385196 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:42.385164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64b8879888-w6k2j_02c3c4b6-1235-455a-bfb4-d38e9e95eb6e/authorino/0.log" Jul 22 17:10:48.312175 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:48.312141 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w6d8p_1e8e5f48-3262-4af7-9282-035d8da49edd/manager/0.log" Jul 22 17:10:48.459299 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:48.459265 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7bf2f_223fb4dd-7b0e-4180-858c-7dd45bb35055/manager/0.log" Jul 22 17:10:48.726770 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:48.726685 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8hgbt_0ba1aada-5cbf-4d18-a198-85165a7e2b1a/server/0.log" Jul 22 17:10:48.859211 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:48.859177 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cxv2p_496a6e1c-f542-4f29-a10b-c5f93ac3b778/manager/0.log" Jul 22 17:10:49.128451 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:49.128426 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb6494f8-hrnd5_5b86b382-ed3c-4952-ad67-bcf3bc5c21a8/copy-manifests/0.log" Jul 22 17:10:49.138132 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:49.138107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb6494f8-hrnd5_5b86b382-ed3c-4952-ad67-bcf3bc5c21a8/manager/0.log" Jul 22 17:10:49.657791 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:49.657761 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d99d56b6d-zmmkh_c769ec4e-3aad-44ef-a2aa-974de95c749d/maas-api/0.log" Jul 22 17:10:50.000836 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:50.000806 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-v9cwj_4bf50400-f6a2-4e70-aa08-e70593892f30/postgres/0.log" Jul 22 17:10:50.855243 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:50.855207 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v_e1902729-34b5-4f83-8bea-f74925cef10d/util/0.log" Jul 22 17:10:50.875636 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:50.875609 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v_e1902729-34b5-4f83-8bea-f74925cef10d/pull/0.log" Jul 22 17:10:50.893254 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:50.893208 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v_e1902729-34b5-4f83-8bea-f74925cef10d/extract/0.log" Jul 22 17:10:51.020415 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.020371 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz_3904e122-3f53-4d22-b31c-f7e9dbb76cc6/util/0.log" Jul 22 17:10:51.032971 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.032950 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz_3904e122-3f53-4d22-b31c-f7e9dbb76cc6/pull/0.log" Jul 22 17:10:51.043239 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.043221 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz_3904e122-3f53-4d22-b31c-f7e9dbb76cc6/extract/0.log" Jul 22 17:10:51.168010 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.167922 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw_ab600634-fcc4-483e-8352-67c95a4fda9a/extract/0.log" Jul 22 17:10:51.185633 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.185609 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw_ab600634-fcc4-483e-8352-67c95a4fda9a/util/0.log" Jul 22 17:10:51.202460 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.202435 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw_ab600634-fcc4-483e-8352-67c95a4fda9a/pull/0.log" Jul 22 17:10:51.318984 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.318952 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq_790d3a30-f612-416c-8bfc-fa90bcaa42a3/extract/0.log" Jul 22 17:10:51.329581 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.329554 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq_790d3a30-f612-416c-8bfc-fa90bcaa42a3/util/0.log" Jul 22 17:10:51.339818 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.339798 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq_790d3a30-f612-416c-8bfc-fa90bcaa42a3/pull/0.log" Jul 22 17:10:51.475747 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.475651 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64b8879888-w6k2j_02c3c4b6-1235-455a-bfb4-d38e9e95eb6e/authorino/0.log" Jul 22 17:10:51.674712 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:51.674682 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ppfns_c77d0333-350a-4653-8287-f3aadf958258/manager/0.log" Jul 22 17:10:52.087912 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:52.087886 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hm6sf_79f7597d-f459-4da9-a7c3-9c8da9a82661/registry-server/0.log" Jul 22 17:10:52.241190 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:52.241161 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-g8nvz_8ed05e73-2d51-4dd3-990e-30a7d01694ad/manager/0.log" Jul 22 17:10:52.515287 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:52.515257 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q744m_961ac820-3682-452a-a58b-d920d2bf1376/manager/0.log" Jul 22 17:10:53.179769 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:53.179736 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788547964b-9bvkq_1813ab2b-baf9-423a-abd1-e3455c6a121c/kube-auth-proxy/0.log" Jul 22 17:10:53.577026 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:53.576999 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xtpcr_a92d4bce-d55a-4967-a42e-62d9f8bd7fae/payload-pre-processing/0.log" Jul 22 17:10:53.884714 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:53.884620 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54fcd77c68-6z9c2_29ad3989-b8cf-44f8-99fa-05eaba47aa39/router/0.log" Jul 22 17:10:54.279029 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:54.278996 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tn6p8_6525da05-8827-4a3a-a267-a21d83b370c7/main/0.log" Jul 22 17:10:54.654685 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:10:54.654600 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg995d_30a64d32-758d-4940-87e3-2887ce4c70c8/main/0.log" Jul 22 17:11:01.453183 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453146 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dg96g/must-gather-b4qwn"] Jul 22 17:11:01.453811 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453790 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefaaf8c-d7be-432b-afc1-a53e221bb98b" containerName="maas-api" Jul 22 17:11:01.453811 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453812 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefaaf8c-d7be-432b-afc1-a53e221bb98b" containerName="maas-api" Jul 22 17:11:01.453966 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453824 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13543bf7-aa28-4608-96d2-406dc7fa8d28" containerName="maas-api" Jul 22 17:11:01.453966 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453832 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13543bf7-aa28-4608-96d2-406dc7fa8d28" containerName="maas-api" Jul 22 17:11:01.453966 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453859 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" containerName="test-curl-7142-140328263718928" Jul 22 17:11:01.453966 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.453869 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" containerName="test-curl-7142-140328263718928" Jul 22 17:11:01.454173 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.454021 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fdf01a3-cabe-47ad-aa80-2f1a483c30b6" containerName="test-curl-7142-140328263718928" Jul 22 17:11:01.454173 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.454037 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13543bf7-aa28-4608-96d2-406dc7fa8d28" containerName="maas-api" Jul 22 17:11:01.454173 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.454048 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefaaf8c-d7be-432b-afc1-a53e221bb98b" containerName="maas-api" Jul 22 17:11:01.457433 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.457386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.460046 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.460017 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dg96g\"/\"default-dockercfg-jwkq6\"" Jul 22 17:11:01.460177 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.460082 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dg96g\"/\"kube-root-ca.crt\"" Jul 22 17:11:01.460177 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.460132 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dg96g\"/\"openshift-service-ca.crt\"" Jul 22 17:11:01.465206 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.465174 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dg96g/must-gather-b4qwn"] Jul 22 17:11:01.513993 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.513953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c2a2fd8c-d6d6-4319-bcd0-25ad1941413d-must-gather-output\") pod \"must-gather-b4qwn\" (UID: \"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d\") " pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.513993 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.513997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nnd\" (UniqueName: \"kubernetes.io/projected/c2a2fd8c-d6d6-4319-bcd0-25ad1941413d-kube-api-access-69nnd\") pod \"must-gather-b4qwn\" (UID: \"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d\") " pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.614931 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.614899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c2a2fd8c-d6d6-4319-bcd0-25ad1941413d-must-gather-output\") pod \"must-gather-b4qwn\" (UID: \"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d\") " pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.614931 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.614941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nnd\" (UniqueName: \"kubernetes.io/projected/c2a2fd8c-d6d6-4319-bcd0-25ad1941413d-kube-api-access-69nnd\") pod \"must-gather-b4qwn\" (UID: \"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d\") " pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.615299 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.615278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c2a2fd8c-d6d6-4319-bcd0-25ad1941413d-must-gather-output\") pod \"must-gather-b4qwn\" (UID: \"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d\") " pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.623919 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.623886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nnd\" (UniqueName: \"kubernetes.io/projected/c2a2fd8c-d6d6-4319-bcd0-25ad1941413d-kube-api-access-69nnd\") pod \"must-gather-b4qwn\" (UID: \"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d\") " pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.768235 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.768204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dg96g/must-gather-b4qwn" Jul 22 17:11:01.910679 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:01.910651 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dg96g/must-gather-b4qwn"] Jul 22 17:11:01.911986 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:11:01.911952 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2a2fd8c_d6d6_4319_bcd0_25ad1941413d.slice/crio-6f56178f8ff1dda66c53c840f1aa8bc0d84d2d933d99df10ef1b5d39370b03a8 WatchSource:0}: Error finding container 6f56178f8ff1dda66c53c840f1aa8bc0d84d2d933d99df10ef1b5d39370b03a8: Status 404 returned error can't find the container with id 6f56178f8ff1dda66c53c840f1aa8bc0d84d2d933d99df10ef1b5d39370b03a8 Jul 22 17:11:02.679937 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:02.679889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg96g/must-gather-b4qwn" event={"ID":"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d","Type":"ContainerStarted","Data":"6f56178f8ff1dda66c53c840f1aa8bc0d84d2d933d99df10ef1b5d39370b03a8"} Jul 22 17:11:03.688939 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:03.688893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg96g/must-gather-b4qwn" event={"ID":"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d","Type":"ContainerStarted","Data":"1c1d479be2519345eb0a37ebff4d806fbbaff066e88e351d9d6887e84bb3e794"} Jul 22 17:11:03.689470 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:03.688954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg96g/must-gather-b4qwn" event={"ID":"c2a2fd8c-d6d6-4319-bcd0-25ad1941413d","Type":"ContainerStarted","Data":"04cbf6aa62d0813d3aef9083555d87a786b7f33684e04cbf70ec45791e5c9287"} Jul 22 17:11:03.705493 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:03.705435 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dg96g/must-gather-b4qwn" podStartSLOduration=1.759870524 podStartE2EDuration="2.705418033s" totalImagesPullingTime="945.547509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:11:03.703055368 +0000 UTC m=+3021.283149266" watchObservedRunningTime="2026-07-22 17:11:03.705418033 +0000 UTC m=+3021.285511930" Jul 22 17:11:04.577325 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:04.577290 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l2gk6_269497b7-5123-4d80-8735-471d64f1e566/global-pull-secret-syncer/0.log" Jul 22 17:11:04.751175 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:04.751149 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sq52k_f0a79340-0d73-44bb-9205-2a863050ec2a/konnectivity-agent/0.log" Jul 22 17:11:04.966110 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:04.966022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-136.ec2.internal_150be6474fc5a2b8d6c7faceb3ac3985/haproxy/0.log" Jul 22 17:11:09.326276 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.326211 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v_e1902729-34b5-4f83-8bea-f74925cef10d/extract/0.log" Jul 22 17:11:09.353761 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.353731 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v_e1902729-34b5-4f83-8bea-f74925cef10d/util/0.log" Jul 22 17:11:09.411307 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.411278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759fpl6v_e1902729-34b5-4f83-8bea-f74925cef10d/pull/0.log" Jul 22 17:11:09.445107 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.445019 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz_3904e122-3f53-4d22-b31c-f7e9dbb76cc6/extract/0.log" Jul 22 17:11:09.480509 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.480483 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz_3904e122-3f53-4d22-b31c-f7e9dbb76cc6/util/0.log" Jul 22 17:11:09.508067 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.508037 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08bgrz_3904e122-3f53-4d22-b31c-f7e9dbb76cc6/pull/0.log" Jul 22 17:11:09.536411 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.536324 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw_ab600634-fcc4-483e-8352-67c95a4fda9a/extract/0.log" Jul 22 17:11:09.562553 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.562496 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw_ab600634-fcc4-483e-8352-67c95a4fda9a/util/0.log" Jul 22 17:11:09.607424 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.607263 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bhbcw_ab600634-fcc4-483e-8352-67c95a4fda9a/pull/0.log" Jul 22 17:11:09.717873 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.717848 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq_790d3a30-f612-416c-8bfc-fa90bcaa42a3/extract/0.log" Jul 22 17:11:09.790752 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.790726 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq_790d3a30-f612-416c-8bfc-fa90bcaa42a3/util/0.log" Jul 22 17:11:09.858843 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:09.858748 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hn9rq_790d3a30-f612-416c-8bfc-fa90bcaa42a3/pull/0.log" Jul 22 17:11:10.803532 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:10.803448 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64b8879888-w6k2j_02c3c4b6-1235-455a-bfb4-d38e9e95eb6e/authorino/0.log" Jul 22 17:11:10.878106 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:10.878046 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ppfns_c77d0333-350a-4653-8287-f3aadf958258/manager/0.log" Jul 22 17:11:11.006907 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:11.006877 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hm6sf_79f7597d-f459-4da9-a7c3-9c8da9a82661/registry-server/0.log" Jul 22 17:11:11.142647 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:11.142564 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-g8nvz_8ed05e73-2d51-4dd3-990e-30a7d01694ad/manager/0.log" Jul 22 17:11:11.360428 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:11.357267 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q744m_961ac820-3682-452a-a58b-d920d2bf1376/manager/0.log" Jul 22 17:11:12.782896 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:12.782863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/alertmanager/0.log" Jul 22 17:11:12.815139 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:12.815107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/config-reloader/0.log" Jul 22 17:11:12.874977 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:12.874948 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/kube-rbac-proxy-web/0.log" Jul 22 17:11:12.918377 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:12.918338 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/kube-rbac-proxy/0.log" Jul 22 17:11:12.991235 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:12.991197 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/kube-rbac-proxy-metric/0.log" Jul 22 17:11:13.017932 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.017892 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/prom-label-proxy/0.log" Jul 22 17:11:13.055712 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.055629 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f716828-080d-4e83-b272-17d664e7a8eb/init-config-reloader/0.log" Jul 22 17:11:13.113345 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.113309 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-lj4xf_64f5290c-94c5-4127-816e-9163bb7034a1/cluster-monitoring-operator/0.log" Jul 22 17:11:13.274813 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.274783 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-f49vh_14523a9d-65dc-4eb8-ab71-be5833fa978d/monitoring-plugin/0.log" Jul 22 17:11:13.303467 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.303427 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8x8ks_c6162922-fec2-43bf-a829-85fdf6b54207/node-exporter/0.log" Jul 22 17:11:13.325298 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.325188 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8x8ks_c6162922-fec2-43bf-a829-85fdf6b54207/kube-rbac-proxy/0.log" Jul 22 17:11:13.367197 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.367172 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8x8ks_c6162922-fec2-43bf-a829-85fdf6b54207/init-textfile/0.log" Jul 22 17:11:13.580019 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.579940 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-5t46n_48b032e6-40fc-418e-9041-1e0928a33eec/kube-rbac-proxy-main/0.log" Jul 22 17:11:13.607227 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.607197 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-5t46n_48b032e6-40fc-418e-9041-1e0928a33eec/kube-rbac-proxy-self/0.log" Jul 22 17:11:13.629282 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.629254 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-5t46n_48b032e6-40fc-418e-9041-1e0928a33eec/openshift-state-metrics/0.log" Jul 22 17:11:13.926149 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.926062 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9cdf7b7c-jrtwz_7b76e213-2523-49ea-8743-320cbde976a0/telemeter-client/0.log" Jul 22 17:11:13.972163 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:13.972131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9cdf7b7c-jrtwz_7b76e213-2523-49ea-8743-320cbde976a0/reload/0.log" Jul 22 17:11:14.020009 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.019979 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9cdf7b7c-jrtwz_7b76e213-2523-49ea-8743-320cbde976a0/kube-rbac-proxy/0.log" Jul 22 17:11:14.064808 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.064777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffbd75db4-lnp9q_d0def3e0-1f9b-4983-82aa-2cb760d79c32/thanos-query/0.log" Jul 22 17:11:14.096103 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.096070 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffbd75db4-lnp9q_d0def3e0-1f9b-4983-82aa-2cb760d79c32/kube-rbac-proxy-web/0.log" Jul 22 17:11:14.150277 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.150245 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffbd75db4-lnp9q_d0def3e0-1f9b-4983-82aa-2cb760d79c32/kube-rbac-proxy/0.log" Jul 22 17:11:14.201171 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.200718 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4"] Jul 22 17:11:14.208831 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.208805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.215921 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.215892 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4"] Jul 22 17:11:14.249449 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.249428 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffbd75db4-lnp9q_d0def3e0-1f9b-4983-82aa-2cb760d79c32/prom-label-proxy/0.log" Jul 22 17:11:14.298853 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.298821 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffbd75db4-lnp9q_d0def3e0-1f9b-4983-82aa-2cb760d79c32/kube-rbac-proxy-rules/0.log" Jul 22 17:11:14.353308 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.353274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-podres\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.353492 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.353316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-sys\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.353492 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.353338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-lib-modules\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.353492 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.353418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-proc\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.353603 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.353551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkccf\" (UniqueName: \"kubernetes.io/projected/167411ae-fde2-475b-a91e-87e92e62b36d-kube-api-access-pkccf\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.374121 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.374094 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ffbd75db4-lnp9q_d0def3e0-1f9b-4983-82aa-2cb760d79c32/kube-rbac-proxy-metrics/0.log" Jul 22 17:11:14.454641 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.454553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-podres\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.454641 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.454611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-sys\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.454857 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.454649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-lib-modules\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.454857 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.454682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-proc\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.454857 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.454773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkccf\" (UniqueName: \"kubernetes.io/projected/167411ae-fde2-475b-a91e-87e92e62b36d-kube-api-access-pkccf\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.455370 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.455339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-podres\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.455526 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.455439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-sys\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.455592 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.455544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-lib-modules\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.455671 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.455595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/167411ae-fde2-475b-a91e-87e92e62b36d-proc\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.464778 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.464750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkccf\" (UniqueName: \"kubernetes.io/projected/167411ae-fde2-475b-a91e-87e92e62b36d-kube-api-access-pkccf\") pod \"perf-node-gather-daemonset-qtlm4\" (UID: \"167411ae-fde2-475b-a91e-87e92e62b36d\") " pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.522708 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.522663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:14.703535 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.703507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4"] Jul 22 17:11:14.705678 ip-10-0-133-136 kubenswrapper[2580]: W0722 17:11:14.705610 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod167411ae_fde2_475b_a91e_87e92e62b36d.slice/crio-2cf7a13d00e144323bddaf590be261f54b7b16683a5bd4c76f389e49ba95b617 WatchSource:0}: Error finding container 2cf7a13d00e144323bddaf590be261f54b7b16683a5bd4c76f389e49ba95b617: Status 404 returned error can't find the container with id 2cf7a13d00e144323bddaf590be261f54b7b16683a5bd4c76f389e49ba95b617 Jul 22 17:11:14.829327 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:14.829279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" event={"ID":"167411ae-fde2-475b-a91e-87e92e62b36d","Type":"ContainerStarted","Data":"2cf7a13d00e144323bddaf590be261f54b7b16683a5bd4c76f389e49ba95b617"} Jul 22 17:11:15.840569 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:15.840532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" event={"ID":"167411ae-fde2-475b-a91e-87e92e62b36d","Type":"ContainerStarted","Data":"a6b5e9e07156786473dc49079f593033cae011f3d007a1d7c25f450b43ac6d82"} Jul 22 17:11:15.841246 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:15.841224 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:15.859353 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:15.859302 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" podStartSLOduration=1.859285276 podStartE2EDuration="1.859285276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 17:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:11:15.858695057 +0000 UTC m=+3033.438788956" watchObservedRunningTime="2026-07-22 17:11:15.859285276 +0000 UTC m=+3033.439379222" Jul 22 17:11:16.538634 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:16.538599 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-755ddcb497-6kgl5_b258aea6-8d3c-4adf-ab09-27af9e8f2c18/console/0.log" Jul 22 17:11:16.580911 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:16.580876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-722k6_2e056b28-ef60-4e85-8c47-f17c5e01ed35/download-server/0.log" Jul 22 17:11:18.270502 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:18.270469 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5b4ll_21622b2f-0c96-4f74-bc80-832288568931/dns/0.log" Jul 22 17:11:18.374443 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:18.374414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5b4ll_21622b2f-0c96-4f74-bc80-832288568931/kube-rbac-proxy/0.log" Jul 22 17:11:18.537609 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:18.537533 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmmv2_8e375cc0-38fb-4257-8a70-bb24353d6f64/dns-node-resolver/0.log" Jul 22 17:11:19.153580 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:19.153549 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69c54c58fd-zhd9r_0ab3b9f1-342f-4985-a599-051e115a06a0/registry/0.log" Jul 22 17:11:19.485759 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:19.485672 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqpjx_826c741a-501c-4c72-a2e9-a793749a666f/node-ca/0.log" Jul 22 17:11:20.809644 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:20.809616 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788547964b-9bvkq_1813ab2b-baf9-423a-abd1-e3455c6a121c/kube-auth-proxy/0.log" Jul 22 17:11:21.014208 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:21.014168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xtpcr_a92d4bce-d55a-4967-a42e-62d9f8bd7fae/payload-pre-processing/0.log" Jul 22 17:11:21.136027 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:21.135940 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54fcd77c68-6z9c2_29ad3989-b8cf-44f8-99fa-05eaba47aa39/router/0.log" Jul 22 17:11:21.930484 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:21.930449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7x4pm_e05bcdb7-ae0b-48e3-b8e1-0b47afaa6664/serve-healthcheck-canary/0.log" Jul 22 17:11:22.500774 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:22.500737 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-p9kbk_e78842c1-409e-4a48-a689-901e7f172152/insights-operator/0.log" Jul 22 17:11:22.501853 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:22.501832 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-p9kbk_e78842c1-409e-4a48-a689-901e7f172152/insights-operator/1.log" Jul 22 17:11:22.672854 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:22.672808 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v89rw_bff7c473-0eea-4e23-85f8-0e7162fe924c/kube-rbac-proxy/0.log" Jul 22 17:11:22.706782 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:22.706756 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v89rw_bff7c473-0eea-4e23-85f8-0e7162fe924c/exporter/0.log" Jul 22 17:11:22.787788 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:22.787711 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v89rw_bff7c473-0eea-4e23-85f8-0e7162fe924c/extractor/0.log" Jul 22 17:11:22.872989 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:22.872956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dg96g/perf-node-gather-daemonset-qtlm4" Jul 22 17:11:24.739608 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:24.739584 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788547964b-9bvkq_1813ab2b-baf9-423a-abd1-e3455c6a121c/kube-auth-proxy/0.log" Jul 22 17:11:24.870195 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:24.870164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xtpcr_a92d4bce-d55a-4967-a42e-62d9f8bd7fae/payload-pre-processing/0.log" Jul 22 17:11:24.925525 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:24.925474 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54fcd77c68-6z9c2_29ad3989-b8cf-44f8-99fa-05eaba47aa39/router/0.log" Jul 22 17:11:25.250272 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:25.250234 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w6d8p_1e8e5f48-3262-4af7-9282-035d8da49edd/manager/0.log" Jul 22 17:11:25.485933 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:25.485876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7bf2f_223fb4dd-7b0e-4180-858c-7dd45bb35055/manager/0.log" Jul 22 17:11:25.918646 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:25.918619 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8hgbt_0ba1aada-5cbf-4d18-a198-85165a7e2b1a/server/0.log" Jul 22 17:11:26.027230 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:26.027195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cxv2p_496a6e1c-f542-4f29-a10b-c5f93ac3b778/manager/0.log" Jul 22 17:11:26.186434 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:26.186331 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb6494f8-hrnd5_5b86b382-ed3c-4952-ad67-bcf3bc5c21a8/manager/0.log" Jul 22 17:11:26.219918 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:26.219889 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb6494f8-hrnd5_5b86b382-ed3c-4952-ad67-bcf3bc5c21a8/copy-manifests/0.log" Jul 22 17:11:26.397167 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:26.397138 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788547964b-9bvkq_1813ab2b-baf9-423a-abd1-e3455c6a121c/kube-auth-proxy/0.log" Jul 22 17:11:26.525242 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:26.525193 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xtpcr_a92d4bce-d55a-4967-a42e-62d9f8bd7fae/payload-pre-processing/0.log" Jul 22 17:11:26.575051 ip-10-0-133-136 kubenswrapper[2580]: I0722 17:11:26.574999 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54fcd77c68-6z9c2_29ad3989-b8cf-44f8-99fa-05eaba47aa39/router/0.log"