Jul 14 19:44:24.388544 ip-10-0-140-43 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 19:44:24.388690 ip-10-0-140-43 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 19:44:24.388957 ip-10-0-140-43 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 19:44:24.389369 ip-10-0-140-43 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 19:44:34.611823 ip-10-0-140-43 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 19:44:34.611839 ip-10-0-140-43 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bef0cc0053af49ad996f23d7b0508bca -- Jul 14 19:46:32.913083 ip-10-0-140-43 systemd[1]: Starting Kubernetes Kubelet... Jul 14 19:46:33.246510 ip-10-0-140-43 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:46:33.246510 ip-10-0-140-43 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 19:46:33.246510 ip-10-0-140-43 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:46:33.246510 ip-10-0-140-43 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 19:46:33.246510 ip-10-0-140-43 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:46:33.248295 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.248200 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 19:46:33.253029 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253011 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:33.253029 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253028 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253031 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253036 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253040 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253042 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253046 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253050 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253054 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253057 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253060 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253063 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253066 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253070 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253072 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253075 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253077 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253080 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253087 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:33.253095 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253090 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253100 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253104 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253107 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253110 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253112 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253115 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253118 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253120 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253122 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253125 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253127 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253130 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253133 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253136 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253138 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253140 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253143 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253146 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253149 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:33.253496 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253151 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253154 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253156 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253158 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253161 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253163 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253166 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253168 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253171 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253173 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253176 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253178 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253180 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253183 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253187 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253190 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253193 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253196 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253198 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253201 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:33.254005 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253203 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253206 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253208 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253210 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253213 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253215 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253218 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253220 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253223 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253226 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253228 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253231 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253234 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253236 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253239 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253241 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253244 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253246 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253248 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253251 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:33.254497 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253253 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253255 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253260 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253263 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253266 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253269 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253271 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253274 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253277 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253280 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253282 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253285 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253287 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253290 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253292 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253295 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253297 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253300 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253302 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.253305 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:33.254983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254011 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254018 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254021 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254025 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254029 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254031 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254034 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254036 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254039 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254042 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254044 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254047 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254049 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254052 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254054 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254057 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254061 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254064 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254067 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:33.255445 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254070 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254073 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254075 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254078 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254080 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254083 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254085 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254088 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254091 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254094 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254096 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254099 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254101 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254104 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254107 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254109 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254112 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254115 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254118 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254120 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:33.255953 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254123 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254126 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254128 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254130 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254133 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254135 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254140 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254143 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254146 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254149 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254152 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254155 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254157 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254160 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254162 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254165 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254167 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254170 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254172 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:33.256439 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254176 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254178 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254181 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254183 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254186 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254188 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254191 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254193 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254195 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254198 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254201 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254204 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254207 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254209 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254212 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254215 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254220 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254223 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254225 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254228 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:33.256948 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254230 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254233 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254235 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254238 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254240 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254242 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254245 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254247 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254250 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254252 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254256 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254259 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254262 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254265 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254268 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254271 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254273 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254276 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254279 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:33.257428 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254281 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254284 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254652 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254657 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254662 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254666 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254669 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254672 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254676 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254682 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254685 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254688 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254691 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254694 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254697 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254699 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254702 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254705 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254708 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254711 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:33.257895 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254714 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254716 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254719 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254722 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254738 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254741 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254744 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254747 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254750 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254752 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254754 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254757 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254760 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254762 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254765 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254768 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254770 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254774 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254776 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254779 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:33.258354 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254781 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254786 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254788 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254791 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254793 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254796 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254798 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254801 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254803 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254806 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254809 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254811 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254813 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254816 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254818 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254821 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254823 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254826 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254829 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254831 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:33.258847 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254833 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254836 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254838 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254842 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254845 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254847 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254849 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254852 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254854 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254856 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254859 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254862 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254864 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254868 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254870 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254873 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254875 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254877 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254880 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254882 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:33.259329 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254886 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254888 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254891 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254893 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254896 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254898 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254900 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254903 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254905 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254908 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254910 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254912 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254914 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254917 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254919 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254922 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254924 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254926 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254929 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:33.259820 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254932 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.254934 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255003 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255011 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255018 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255022 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255029 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255033 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255037 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255041 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255044 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255047 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255050 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255054 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255057 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255060 2553 flags.go:64] FLAG: --cgroup-root="" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255063 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255066 2553 flags.go:64] FLAG: --client-ca-file="" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255069 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255072 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255076 2553 flags.go:64] FLAG: --cluster-domain="" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255078 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255081 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255084 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 19:46:33.260257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255088 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255091 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255094 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255097 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255101 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255103 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255106 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255109 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255112 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255115 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255118 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255121 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255123 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255127 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255130 2553 flags.go:64] FLAG: --enable-server="true" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255133 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255138 2553 flags.go:64] FLAG: --event-burst="100" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255141 2553 flags.go:64] FLAG: --event-qps="50" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255144 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255147 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255149 2553 flags.go:64] FLAG: --eviction-hard="" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255158 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255168 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255171 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255174 2553 flags.go:64] FLAG: --eviction-soft="" Jul 14 19:46:33.260856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255176 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255179 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255188 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255192 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255195 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255198 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255200 2553 flags.go:64] FLAG: --feature-gates="" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255204 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255207 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255210 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255213 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255216 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255219 2553 flags.go:64] FLAG: --help="false" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255222 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-140-43.ec2.internal" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255225 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255228 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255231 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255234 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255237 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255240 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255242 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255246 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255249 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 19:46:33.261467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255252 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255255 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255258 2553 flags.go:64] FLAG: --kube-reserved="" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255261 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255263 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255268 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255270 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255273 2553 flags.go:64] FLAG: --lock-file="" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255276 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255278 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255281 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255286 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255290 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255293 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255295 2553 flags.go:64] FLAG: --logging-format="text" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255298 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255302 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255305 2553 flags.go:64] FLAG: --manifest-url="" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255307 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255311 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255314 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255318 2553 flags.go:64] FLAG: --max-pods="110" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255321 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255324 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255327 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 19:46:33.262088 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255329 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255332 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255336 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255338 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255346 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255349 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255352 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255355 2553 flags.go:64] FLAG: --pod-cidr="" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255359 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255365 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255368 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255371 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255375 2553 flags.go:64] FLAG: --port="10250" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255378 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255380 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eb35f4c25b839d22" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255383 2553 flags.go:64] FLAG: --qos-reserved="" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255386 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255389 2553 flags.go:64] FLAG: --register-node="true" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255392 2553 flags.go:64] FLAG: --register-with-taints="" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255395 2553 flags.go:64] FLAG: --registry-burst="10" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255398 2553 flags.go:64] FLAG: --registry-qps="5" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255400 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255403 2553 flags.go:64] FLAG: --reserved-memory="" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255407 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255410 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 19:46:33.262675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255412 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255415 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255418 2553 flags.go:64] FLAG: --runonce="false" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255421 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255423 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255426 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255429 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255431 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255434 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255438 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255441 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255443 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255446 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255450 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255454 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255457 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255460 2553 flags.go:64] FLAG: --system-cgroups="" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255462 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255468 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255471 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255474 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255478 2553 flags.go:64] FLAG: --tls-min-version="" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255481 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255483 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255486 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 19:46:33.263388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255489 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255492 2553 flags.go:64] FLAG: --v="2" Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255496 2553 flags.go:64] FLAG: --version="false" Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255500 2553 flags.go:64] FLAG: --vmodule="" Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255503 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.255506 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255617 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255620 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255624 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255627 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255630 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255633 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255635 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255638 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255641 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255644 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255646 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255649 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255651 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255654 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255658 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:33.264045 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255660 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255663 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255666 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255668 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255671 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255675 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255677 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255680 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255683 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255685 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255688 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255690 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255693 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255696 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255698 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255701 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255703 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255705 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255708 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:33.264911 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255710 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255713 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255715 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255717 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255720 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255736 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255742 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255745 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255749 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255753 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255756 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255759 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255763 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255765 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255768 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255771 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255774 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255777 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255781 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255783 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:33.265474 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255786 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255788 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255791 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255793 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255796 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255798 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255801 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255803 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255806 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255809 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255813 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255816 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255819 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255822 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255825 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255828 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255830 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255833 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255835 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:33.266074 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255838 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255840 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255842 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255845 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255847 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255851 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255854 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255856 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255859 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255863 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255866 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255870 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255873 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255876 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255879 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255882 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255885 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255888 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255890 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:33.266509 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255893 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255895 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255898 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255900 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255903 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255905 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.255907 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.256380 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.262711 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.262740 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.263484 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.263491 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263547 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263551 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263555 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:33.266972 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263558 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263561 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263564 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263566 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263569 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263572 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263575 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263578 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263580 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263583 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263585 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263588 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263591 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263593 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263596 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263598 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263601 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263603 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263605 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:33.267352 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263608 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263610 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263612 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263615 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263617 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263620 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263622 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263627 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263630 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263632 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263635 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263637 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263640 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263642 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263645 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263647 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263650 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263652 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263655 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:33.267813 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263659 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263663 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263666 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263668 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263671 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263673 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263676 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263678 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263681 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263683 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263685 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263688 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263690 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263693 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263695 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263699 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263702 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263705 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263708 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263711 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:33.268266 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263716 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263719 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263722 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263736 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263739 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263743 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263745 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263748 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263750 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263753 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263755 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263758 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263761 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263763 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263767 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263771 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263773 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263776 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263779 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:33.268758 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263782 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263785 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263788 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263790 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263793 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263796 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263799 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263801 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263804 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263806 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263808 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263811 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263813 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263816 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263819 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263822 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263825 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263828 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:33.269206 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.263830 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.263836 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264898 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264918 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264925 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264930 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264934 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264938 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264943 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264954 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264959 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264963 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264967 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264972 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264976 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:33.269676 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264980 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264985 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264989 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264993 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.264997 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265001 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265012 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265018 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265023 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265027 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265031 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265035 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265039 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265043 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265047 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265052 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265056 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265060 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265064 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:33.270121 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265074 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265080 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265085 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265092 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265097 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265102 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265106 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265111 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265115 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265121 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265126 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265130 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265139 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265143 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265147 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265150 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265154 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265158 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:33.270564 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265162 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265169 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265173 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265176 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265180 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265185 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265190 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265199 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265203 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265206 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265209 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265212 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265215 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265217 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265221 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265226 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265231 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265236 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265240 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265244 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265248 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:33.271000 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265252 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265261 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265265 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265280 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265286 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265290 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265405 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265411 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265414 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265417 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265419 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265422 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265425 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265429 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265431 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265434 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265436 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265439 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265442 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265445 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:33.271490 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265448 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265450 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265453 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265455 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265457 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265460 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265463 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:33.265465 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.265472 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.265600 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.267381 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.268442 2553 server.go:1019] "Starting client certificate rotation" Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.268544 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:46:33.271983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.268572 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:46:33.272297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.270530 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:46:33.275022 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:33.275008 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:46:34.822479 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.822460 2553 log.go:25] "Validated CRI v1 runtime API" Jul 14 19:46:34.824332 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.824318 2553 log.go:25] "Validated CRI v1 image API" Jul 14 19:46:34.825695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.825680 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 19:46:34.831224 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.831202 2553 fs.go:135] Filesystem UUIDs: map[59616996-0120-44eb-bc15-34b292cf5c85:/dev/nvme0n1p4 596a58f8-1484-4d37-8779-54a3ed3c14e8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 14 19:46:34.831287 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.831222 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 19:46:34.833010 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.832995 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:46:34.837830 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.837728 2553 manager.go:216] Machine: {Timestamp:2026-07-14 19:46:34.835685203 +0000 UTC m=+0.284311639 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200378 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279c8ed786d111d7d7eeea6ae3acdd SystemUUID:ec279c8e-d786-d111-d7d7-eeea6ae3acdd BootID:bef0cc00-53af-49ad-996f-23d7b0508bca Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:df:ec:e9:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:df:ec:e9:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:ae:0a:bd:f6:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 19:46:34.837830 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.837825 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 19:46:34.837934 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.837902 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 19:46:34.840147 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.840122 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 19:46:34.840277 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.840149 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 19:46:34.840735 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.840726 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 19:46:34.840763 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.840737 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 19:46:34.840763 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.840744 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:46:34.841417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.841407 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:46:34.842312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.842303 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 19:46:34.844582 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844571 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:46:34.844867 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844857 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 19:46:34.844897 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844889 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 14 19:46:34.844923 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844903 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 19:46:34.844923 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844915 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 19:46:34.844974 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844925 2553 kubelet.go:398] "Adding apiserver pod source" Jul 14 19:46:34.844974 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.844935 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 19:46:34.845317 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.845306 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:46:34.845357 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.845324 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:46:34.848093 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.848058 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 19:46:34.849607 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849589 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 19:46:34.849648 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849624 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 19:46:34.849798 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849784 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 19:46:34.849838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849805 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 19:46:34.849838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849815 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 19:46:34.849838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849823 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 19:46:34.849838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849835 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 19:46:34.849947 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.849843 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 19:46:34.850287 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.850278 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 19:46:34.850322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.850288 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 19:46:34.850322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.850295 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 19:46:34.850322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.850300 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 19:46:34.850322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.850308 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 19:46:34.850322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.850315 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 19:46:34.851648 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.851638 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 19:46:34.851648 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.851648 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 19:46:34.854971 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.854958 2553 server.go:1268] "Started kubelet" Jul 14 19:46:34.855125 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.855105 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 19:46:34.855225 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.855057 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 19:46:34.855278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.855210 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 19:46:34.855278 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.855246 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 19:46:34.855278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.855271 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 19:46:34.855461 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.855438 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 19:46:34.856292 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.856276 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 19:46:34.856557 ip-10-0-140-43 systemd[1]: Started Kubernetes Kubelet. Jul 14 19:46:34.857648 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.857634 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 14 19:46:34.860679 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.859860 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18c23fa863e1aa49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-07-14 19:46:34.854926921 +0000 UTC m=+0.303553359,LastTimestamp:2026-07-14 19:46:34.854926921 +0000 UTC m=+0.303553359,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}" Jul 14 19:46:34.861183 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861165 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 19:46:34.861261 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861178 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 19:46:34.861261 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.861242 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:34.861696 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861680 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 19:46:34.861814 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861729 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 19:46:34.861883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861835 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 19:46:34.861931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861917 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 14 19:46:34.861931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.861924 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 14 19:46:34.862164 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862147 2553 factory.go:55] Registering systemd factory Jul 14 19:46:34.862164 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862165 2553 factory.go:223] Registration of the systemd container factory successfully Jul 14 19:46:34.862270 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.862148 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 19:46:34.862394 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862371 2553 factory.go:153] Registering CRI-O factory Jul 14 19:46:34.862432 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862396 2553 factory.go:223] Registration of the crio container factory successfully Jul 14 19:46:34.862468 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862454 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 19:46:34.862503 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862471 2553 factory.go:103] Registering Raw factory Jul 14 19:46:34.862503 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862485 2553 manager.go:1196] Started watching for new ooms in manager Jul 14 19:46:34.862868 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.862858 2553 manager.go:349] Starting recovery of all containers Jul 14 19:46:34.869275 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.869247 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 19:46:34.869388 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.869279 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 19:46:34.873628 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.873456 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7f4b" Jul 14 19:46:34.874189 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.874174 2553 manager.go:354] Recovery completed Jul 14 19:46:34.879759 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.879745 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:34.881594 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.881580 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:34.881664 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.881605 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:34.881664 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.881615 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:34.882078 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882052 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 19:46:34.882169 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882081 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:34.882169 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882085 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 19:46:34.882169 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882130 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:46:34.882639 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882619 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7f4b" Jul 14 19:46:34.882811 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882793 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:34.882883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882822 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:34.882883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.882832 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:34.883304 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.883292 2553 policy_none.go:49] "None policy: Start" Jul 14 19:46:34.883361 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.883310 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 19:46:34.883361 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.883325 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 19:46:34.883622 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.883527 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18c23fa8657890e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-07-14 19:46:34.881593573 +0000 UTC m=+0.330220009,LastTimestamp:2026-07-14 19:46:34.881593573 +0000 UTC m=+0.330220009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}" Jul 14 19:46:34.884364 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.884351 2553 policy_none.go:47] "Start" Jul 14 19:46:34.926708 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.926689 2553 manager.go:337] "Starting Device Plugin manager" Jul 14 19:46:34.926837 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.926787 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 19:46:34.926837 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.926801 2553 server.go:87] "Starting device plugin registration server" Jul 14 19:46:34.927132 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.927119 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 19:46:34.927200 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.927134 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 19:46:34.927254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.927236 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 19:46:34.927572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.927344 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 19:46:34.927572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:34.927381 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 19:46:34.927866 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.927842 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 19:46:34.927968 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:34.927901 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.027275 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.027252 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:35.028739 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.028700 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 19:46:35.028833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.028747 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:35.028833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.028773 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:35.028833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.028784 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:35.028833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.028813 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.029997 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.029980 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 19:46:35.030071 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.030033 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 19:46:35.030125 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.030072 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 19:46:35.030125 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.030113 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 19:46:35.031795 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.031778 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:35.036244 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.036203 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.036244 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.036222 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-43.ec2.internal\": node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.081827 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.081802 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.130643 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.130599 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"] Jul 14 19:46:35.130760 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.130702 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:35.132283 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.132263 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:35.132396 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.132292 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:35.132396 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.132302 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:35.133420 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.133407 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:35.133563 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.133547 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.133620 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.133585 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:35.134124 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.134102 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:35.134124 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.134123 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:35.134254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.134129 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:35.134254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.134143 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:35.134254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.134154 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:35.134254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.134143 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:35.135445 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.135430 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.135504 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.135455 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:35.136057 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.136041 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:35.136163 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.136084 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:35.136163 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.136098 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:35.163755 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.163733 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7429f60e293240c012ee36892b0b6e39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"7429f60e293240c012ee36892b0b6e39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.163858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.163759 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7429f60e293240c012ee36892b0b6e39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"7429f60e293240c012ee36892b0b6e39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.163858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.163775 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/452d5b1c76cd4f857f719491897be84c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-43.ec2.internal\" (UID: \"452d5b1c76cd4f857f719491897be84c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.163858 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.163783 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-43.ec2.internal\" not found" node="ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.167154 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.167139 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-43.ec2.internal\" not found" node="ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.182105 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.182082 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.264069 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.264038 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7429f60e293240c012ee36892b0b6e39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"7429f60e293240c012ee36892b0b6e39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.264153 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.263998 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7429f60e293240c012ee36892b0b6e39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"7429f60e293240c012ee36892b0b6e39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.264153 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.264109 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/452d5b1c76cd4f857f719491897be84c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-43.ec2.internal\" (UID: \"452d5b1c76cd4f857f719491897be84c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.264153 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.264128 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7429f60e293240c012ee36892b0b6e39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"7429f60e293240c012ee36892b0b6e39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.264256 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.264170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7429f60e293240c012ee36892b0b6e39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"7429f60e293240c012ee36892b0b6e39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.264256 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.264180 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/452d5b1c76cd4f857f719491897be84c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-43.ec2.internal\" (UID: \"452d5b1c76cd4f857f719491897be84c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.283125 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.283099 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.383998 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.383921 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.465334 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.465295 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.469916 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.469897 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.484812 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.484789 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.585337 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.585301 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found" Jul 14 19:46:35.673012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.672935 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:35.761648 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.761616 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.776298 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.776270 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:46:35.778035 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.778020 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" Jul 14 19:46:35.784671 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.784656 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:46:35.804702 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.804672 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 19:46:35.804874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.804856 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:46:35.804928 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.804878 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:46:35.845974 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.845941 2553 apiserver.go:52] "Watching apiserver" Jul 14 19:46:35.861397 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.861372 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 19:46:35.867432 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.866642 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:35.869752 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.869736 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 19:46:35.870049 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.870031 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal","openshift-image-registry/node-ca-fxc4r","openshift-multus/multus-additional-cni-plugins-hp9tz","openshift-multus/network-metrics-daemon-hj4v9","openshift-network-diagnostics/network-check-target-8fbqx","openshift-network-operator/iptables-alerter-qqgd9","kube-system/global-pull-secret-syncer-26pgr","kube-system/konnectivity-agent-2zjw5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp","openshift-cluster-node-tuning-operator/tuned-dbt6j","openshift-dns/node-resolver-njn4q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal","openshift-multus/multus-gnt9t","openshift-ovn-kubernetes/ovnkube-node-ddrpt"] Jul 14 19:46:35.871615 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.871601 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:35.872538 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.872517 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.873800 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.873785 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.873800 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.873795 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:35.873938 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.873840 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:35.874097 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874082 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 19:46:35.874347 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874334 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5nfkz\"" Jul 14 19:46:35.874547 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874536 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.874733 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874718 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.874819 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874805 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.874920 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874874 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 19:46:35.874968 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.874923 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:35.875017 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.874972 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:35.875225 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.875210 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4c6r7\"" Jul 14 19:46:35.875267 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.875212 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 19:46:35.875267 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.875239 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 19:46:35.875858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.875842 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:35.876801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.876783 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:46:35.876985 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.876971 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:35.877050 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:35.877018 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:35.878010 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.877995 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:35.878102 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.878018 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 19:46:35.878102 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.878018 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x49s5\"" Jul 14 19:46:35.878202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.878148 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.878202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.878157 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.879033 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.879017 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.879906 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.879892 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 19:46:35.880169 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.880154 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 19:46:35.880239 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.880224 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.880239 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.880158 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qjvdn\"" Jul 14 19:46:35.880944 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.880928 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g4tpb\"" Jul 14 19:46:35.881008 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.880996 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.881303 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.881288 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 19:46:35.881390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.881378 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:35.881490 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.881476 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.882118 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.882105 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dnwd\"" Jul 14 19:46:35.882171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.882110 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.882499 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.882487 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.882684 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.882670 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.883464 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.883450 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wsd65\"" Jul 14 19:46:35.883533 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.883451 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.883568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.883552 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.884011 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.883993 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.884574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.884557 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tzmln\"" Jul 14 19:46:35.884666 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.884620 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 19:46:35.885163 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.885144 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:41:34 +0000 UTC" deadline="2028-03-22 10:32:02.17280006 +0000 UTC" Jul 14 19:46:35.885220 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.885164 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14798h45m26.287639993s" Jul 14 19:46:35.886032 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.886017 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 19:46:35.887318 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.887303 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 19:46:35.887414 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.887321 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.887414 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.887341 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.887536 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.887511 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 19:46:35.887536 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.887511 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xc9b\"" Jul 14 19:46:35.887658 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.887525 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 19:46:35.906801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.906780 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hmfh" Jul 14 19:46:35.916016 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.915923 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hmfh" Jul 14 19:46:35.922087 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.922054 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:35.962816 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.962792 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 19:46:35.967966 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.967945 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-sys-fs\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.968049 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.967975 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-modprobe-d\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968049 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.967994 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-tuned\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968049 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968009 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwfpv\" (UniqueName: \"kubernetes.io/projected/3cb36332-2f1b-4aee-a15b-951633f6e6cc-kube-api-access-bwfpv\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:35.968049 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-var-lib-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-ovnkube-config\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968106 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c720d8d6-c35c-437f-bcbd-fee227e73d46-agent-certs\") pod \"konnectivity-agent-2zjw5\" (UID: \"c720d8d6-c35c-437f-bcbd-fee227e73d46\") " pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:35.968240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968136 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-lib-modules\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968158 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968182 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-cni-bin\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968225 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-ovnkube-script-lib\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968261 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-netns\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968290 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-sys\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968329 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-etc-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968373 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968403 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8kq\" (UniqueName: \"kubernetes.io/projected/9449ff7f-e1fe-4a43-a747-1091a98dd62c-kube-api-access-ns8kq\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968433 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/437754cb-a459-4410-b75f-8ee2f10721ba-kubelet-config\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:35.968459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968452 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-cni-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968470 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-socket-dir-parent\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-cni-bin\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968533 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljmh\" (UniqueName: \"kubernetes.io/projected/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-kube-api-access-jljmh\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968561 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cnibin\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968600 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968627 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-kubernetes\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968649 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysctl-d\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-slash\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968695 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-log-socket\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968719 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968741 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-hostroot\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968762 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-run-netns\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968787 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-node-log\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/ebae7f13-3078-42b4-b221-aa64593e0bfb-kube-api-access-t8scg\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968828 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysctl-conf\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb36332-2f1b-4aee-a15b-951633f6e6cc-hosts-file\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-systemd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968886 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-ovn\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-env-overrides\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.968964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968945 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zwv\" (UniqueName: \"kubernetes.io/projected/fef8297a-791a-4cfd-9880-3b6510c03642-kube-api-access-f7zwv\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.968983 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-socket-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969006 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-registration-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-device-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37d713aa-4f27-416c-b5cd-55a61898b273-ovn-node-metrics-cert\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969054 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969087 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvjr\" (UniqueName: \"kubernetes.io/projected/07428bc7-8a29-4650-b882-b2ba2fceca25-kube-api-access-shvjr\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969102 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-system-cni-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969116 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-cnibin\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969132 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-kubelet\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysconfig\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969179 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb36332-2f1b-4aee-a15b-951633f6e6cc-tmp-dir\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969207 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969234 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-systemd\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969257 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-kubelet\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969279 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-systemd-units\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.969426 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969303 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969338 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969359 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-k8s-cni-cncf-io\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969399 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-cni-multus\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-daemon-config\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969437 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53011eae-e1ec-4f43-aee9-aa5ac3a52859-host\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969455 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-cni-netd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969474 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-system-cni-dir\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969489 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969507 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/437754cb-a459-4410-b75f-8ee2f10721ba-dbus\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969535 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-os-release\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969568 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-etc-kubernetes\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969593 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53011eae-e1ec-4f43-aee9-aa5ac3a52859-serviceca\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969617 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969640 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969664 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-multus-certs\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969687 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-run\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.970012 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969710 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-var-lib-kubelet\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969732 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4sd\" (UniqueName: \"kubernetes.io/projected/37d713aa-4f27-416c-b5cd-55a61898b273-kube-api-access-2q4sd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969756 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-os-release\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969779 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fef8297a-791a-4cfd-9880-3b6510c03642-iptables-alerter-script\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969802 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c720d8d6-c35c-437f-bcbd-fee227e73d46-konnectivity-ca\") pod \"konnectivity-agent-2zjw5\" (UID: \"c720d8d6-c35c-437f-bcbd-fee227e73d46\") " pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969827 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjh8z\" (UniqueName: \"kubernetes.io/projected/53011eae-e1ec-4f43-aee9-aa5ac3a52859-kube-api-access-tjh8z\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-host\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-conf-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969894 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-cni-binary-copy\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969927 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969943 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fef8297a-791a-4cfd-9880-3b6510c03642-host-slash\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969964 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b529c87-9604-4046-bff6-a0a0a5a60154-tmp\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:35.970541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:35.969982 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pv2z\" (UniqueName: \"kubernetes.io/projected/8b529c87-9604-4046-bff6-a0a0a5a60154-kube-api-access-7pv2z\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.070808 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-systemd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.070808 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-ovn\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-env-overrides\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070841 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zwv\" (UniqueName: \"kubernetes.io/projected/fef8297a-791a-4cfd-9880-3b6510c03642-kube-api-access-f7zwv\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070859 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-socket-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070873 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-registration-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-device-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070902 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-systemd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070911 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37d713aa-4f27-416c-b5cd-55a61898b273-ovn-node-metrics-cert\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070932 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-ovn\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070940 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070978 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-device-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.070991 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvjr\" (UniqueName: \"kubernetes.io/projected/07428bc7-8a29-4650-b882-b2ba2fceca25-kube-api-access-shvjr\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071017 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071027 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-registration-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-system-cni-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071170 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-cnibin\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071191 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-kubelet\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071088 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-socket-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071224 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysconfig\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-cnibin\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071248 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb36332-2f1b-4aee-a15b-951633f6e6cc-tmp-dir\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071254 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-kubelet\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071271 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071286 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysconfig\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071288 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-systemd\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071319 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-kubelet\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071327 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-systemd-units\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071339 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:36.071585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071389 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071394 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-kubelet\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071365 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-systemd\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071397 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-systemd-units\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071429 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-env-overrides\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.071449 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071081 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-system-cni-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071436 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071486 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-k8s-cni-cncf-io\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.071508 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.57148754 +0000 UTC m=+2.020113982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071532 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-k8s-cni-cncf-io\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071540 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb36332-2f1b-4aee-a15b-951633f6e6cc-tmp-dir\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071543 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-cni-multus\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071575 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-cni-multus\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071595 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-daemon-config\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071618 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53011eae-e1ec-4f43-aee9-aa5ac3a52859-host\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071638 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-cni-netd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.072398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071661 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-system-cni-dir\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071707 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/437754cb-a459-4410-b75f-8ee2f10721ba-dbus\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071725 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-os-release\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071705 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53011eae-e1ec-4f43-aee9-aa5ac3a52859-host\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071747 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-etc-kubernetes\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-cni-netd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071764 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/437754cb-a459-4410-b75f-8ee2f10721ba-dbus\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071785 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-etc-kubernetes\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071787 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53011eae-e1ec-4f43-aee9-aa5ac3a52859-serviceca\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-system-cni-dir\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071822 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071847 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071867 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-multus-certs\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-run\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.071893 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071916 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-var-lib-kubelet\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071924 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-run-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.073191 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071939 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4sd\" (UniqueName: \"kubernetes.io/projected/37d713aa-4f27-416c-b5cd-55a61898b273-kube-api-access-2q4sd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.071956 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.571940414 +0000 UTC m=+2.020566861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071894 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-os-release\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071971 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-run\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.071984 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-os-release\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072011 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fef8297a-791a-4cfd-9880-3b6510c03642-iptables-alerter-script\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072020 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-var-lib-kubelet\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072037 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c720d8d6-c35c-437f-bcbd-fee227e73d46-konnectivity-ca\") pod \"konnectivity-agent-2zjw5\" (UID: \"c720d8d6-c35c-437f-bcbd-fee227e73d46\") " pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjh8z\" (UniqueName: \"kubernetes.io/projected/53011eae-e1ec-4f43-aee9-aa5ac3a52859-kube-api-access-tjh8z\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-host\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072124 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-conf-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072146 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-cni-binary-copy\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072171 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fef8297a-791a-4cfd-9880-3b6510c03642-host-slash\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53011eae-e1ec-4f43-aee9-aa5ac3a52859-serviceca\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072218 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b529c87-9604-4046-bff6-a0a0a5a60154-tmp\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072234 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-os-release\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.073993 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072012 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-multus-certs\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072249 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pv2z\" (UniqueName: \"kubernetes.io/projected/8b529c87-9604-4046-bff6-a0a0a5a60154-kube-api-access-7pv2z\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072261 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072277 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-sys-fs\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072312 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-modprobe-d\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072322 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07428bc7-8a29-4650-b882-b2ba2fceca25-sys-fs\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072338 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-tuned\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072362 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwfpv\" (UniqueName: \"kubernetes.io/projected/3cb36332-2f1b-4aee-a15b-951633f6e6cc-kube-api-access-bwfpv\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072387 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-var-lib-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072409 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-ovnkube-config\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072432 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c720d8d6-c35c-437f-bcbd-fee227e73d46-agent-certs\") pod \"konnectivity-agent-2zjw5\" (UID: \"c720d8d6-c35c-437f-bcbd-fee227e73d46\") " pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072453 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-lib-modules\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072476 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072500 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-cni-bin\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072517 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-var-lib-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-ovnkube-script-lib\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072557 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fef8297a-791a-4cfd-9880-3b6510c03642-iptables-alerter-script\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.074781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072566 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-netns\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-sys\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072598 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-modprobe-d\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072616 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-etc-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072616 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072642 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072671 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8kq\" (UniqueName: \"kubernetes.io/projected/9449ff7f-e1fe-4a43-a747-1091a98dd62c-kube-api-access-ns8kq\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072694 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/437754cb-a459-4410-b75f-8ee2f10721ba-kubelet-config\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072719 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-cni-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072735 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c720d8d6-c35c-437f-bcbd-fee227e73d46-konnectivity-ca\") pod \"konnectivity-agent-2zjw5\" (UID: \"c720d8d6-c35c-437f-bcbd-fee227e73d46\") " pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072742 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-socket-dir-parent\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072783 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-cni-bin\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072807 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljmh\" (UniqueName: \"kubernetes.io/projected/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-kube-api-access-jljmh\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072829 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cnibin\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072860 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-kubernetes\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072907 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysctl-d\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.075568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072931 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-slash\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072953 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-log-socket\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072966 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-host\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.072979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-hostroot\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073010 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-conf-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073049 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-ovnkube-config\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073090 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-hostroot\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073140 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-cni-bin\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073542 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-cni-binary-copy\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073590 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-run-netns\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073619 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-node-log\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/ebae7f13-3078-42b4-b221-aa64593e0bfb-kube-api-access-t8scg\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37d713aa-4f27-416c-b5cd-55a61898b273-ovnkube-script-lib\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073691 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-run-netns\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073725 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysctl-conf\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073752 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb36332-2f1b-4aee-a15b-951633f6e6cc-hosts-file\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073855 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb36332-2f1b-4aee-a15b-951633f6e6cc-hosts-file\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.076417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073905 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-sys\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.073958 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-etc-openvswitch\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074158 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fef8297a-791a-4cfd-9880-3b6510c03642-host-slash\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-run-netns\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074253 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-node-log\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074465 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cnibin\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074477 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/437754cb-a459-4410-b75f-8ee2f10721ba-kubelet-config\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074729 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-socket-dir-parent\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074760 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-host-var-lib-cni-bin\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074843 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b529c87-9604-4046-bff6-a0a0a5a60154-tmp\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074879 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-tuned\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074891 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-host-slash\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074895 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-cni-dir\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074958 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37d713aa-4f27-416c-b5cd-55a61898b273-log-socket\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074994 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysctl-conf\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.074998 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9449ff7f-e1fe-4a43-a747-1091a98dd62c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075003 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-kubernetes\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.077312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075083 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-etc-sysctl-d\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.078284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075120 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b529c87-9604-4046-bff6-a0a0a5a60154-lib-modules\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.078284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075273 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37d713aa-4f27-416c-b5cd-55a61898b273-ovn-node-metrics-cert\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.078284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075346 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9449ff7f-e1fe-4a43-a747-1091a98dd62c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.078284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075430 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c720d8d6-c35c-437f-bcbd-fee227e73d46-agent-certs\") pod \"konnectivity-agent-2zjw5\" (UID: \"c720d8d6-c35c-437f-bcbd-fee227e73d46\") " pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:36.078284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.075555 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-multus-daemon-config\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.092555 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.092530 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvjr\" (UniqueName: \"kubernetes.io/projected/07428bc7-8a29-4650-b882-b2ba2fceca25-kube-api-access-shvjr\") pod \"aws-ebs-csi-driver-node-k7frp\" (UID: \"07428bc7-8a29-4650-b882-b2ba2fceca25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.094829 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.094696 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:36.094829 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.094717 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:36.094829 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.094730 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:36.094829 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.094794 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.594777781 +0000 UTC m=+2.043404210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:36.096215 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.096196 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4sd\" (UniqueName: \"kubernetes.io/projected/37d713aa-4f27-416c-b5cd-55a61898b273-kube-api-access-2q4sd\") pod \"ovnkube-node-ddrpt\" (UID: \"37d713aa-4f27-416c-b5cd-55a61898b273\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.096295 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.096214 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljmh\" (UniqueName: \"kubernetes.io/projected/fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8-kube-api-access-jljmh\") pod \"multus-gnt9t\" (UID: \"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8\") " pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.096343 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.096308 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pv2z\" (UniqueName: \"kubernetes.io/projected/8b529c87-9604-4046-bff6-a0a0a5a60154-kube-api-access-7pv2z\") pod \"tuned-dbt6j\" (UID: \"8b529c87-9604-4046-bff6-a0a0a5a60154\") " pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.096688 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.096666 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjh8z\" (UniqueName: \"kubernetes.io/projected/53011eae-e1ec-4f43-aee9-aa5ac3a52859-kube-api-access-tjh8z\") pod \"node-ca-fxc4r\" (UID: \"53011eae-e1ec-4f43-aee9-aa5ac3a52859\") " pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.096799 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.096691 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zwv\" (UniqueName: \"kubernetes.io/projected/fef8297a-791a-4cfd-9880-3b6510c03642-kube-api-access-f7zwv\") pod \"iptables-alerter-qqgd9\" (UID: \"fef8297a-791a-4cfd-9880-3b6510c03642\") " pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.097496 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.097479 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8kq\" (UniqueName: \"kubernetes.io/projected/9449ff7f-e1fe-4a43-a747-1091a98dd62c-kube-api-access-ns8kq\") pod \"multus-additional-cni-plugins-hp9tz\" (UID: \"9449ff7f-e1fe-4a43-a747-1091a98dd62c\") " pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.097807 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.097792 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwfpv\" (UniqueName: \"kubernetes.io/projected/3cb36332-2f1b-4aee-a15b-951633f6e6cc-kube-api-access-bwfpv\") pod \"node-resolver-njn4q\" (UID: \"3cb36332-2f1b-4aee-a15b-951633f6e6cc\") " pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.097906 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.097892 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/ebae7f13-3078-42b4-b221-aa64593e0bfb-kube-api-access-t8scg\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:36.196392 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.196340 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fxc4r" Jul 14 19:46:36.202018 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.201984 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452d5b1c76cd4f857f719491897be84c.slice/crio-4541d2f38ebdc11697405a06ddf34a33d33df11cddfef2e36145f595cee887f6 WatchSource:0}: Error finding container 4541d2f38ebdc11697405a06ddf34a33d33df11cddfef2e36145f595cee887f6: Status 404 returned error can't find the container with id 4541d2f38ebdc11697405a06ddf34a33d33df11cddfef2e36145f595cee887f6 Jul 14 19:46:36.203865 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.203847 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53011eae_e1ec_4f43_aee9_aa5ac3a52859.slice/crio-21003970f75cb3a1ee8bfd9bba2a329017b354e696e5c0d498dab6e05b0fc4e7 WatchSource:0}: Error finding container 21003970f75cb3a1ee8bfd9bba2a329017b354e696e5c0d498dab6e05b0fc4e7: Status 404 returned error can't find the container with id 21003970f75cb3a1ee8bfd9bba2a329017b354e696e5c0d498dab6e05b0fc4e7 Jul 14 19:46:36.206476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.206460 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:46:36.220593 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.220274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" Jul 14 19:46:36.226873 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.226851 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9449ff7f_e1fe_4a43_a747_1091a98dd62c.slice/crio-aa86e4deb1b0feed2fab5b8b3dbf4fe33ffae31e766abe7f63aad5ad8ebd86e6 WatchSource:0}: Error finding container aa86e4deb1b0feed2fab5b8b3dbf4fe33ffae31e766abe7f63aad5ad8ebd86e6: Status 404 returned error can't find the container with id aa86e4deb1b0feed2fab5b8b3dbf4fe33ffae31e766abe7f63aad5ad8ebd86e6 Jul 14 19:46:36.242714 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.242696 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qqgd9" Jul 14 19:46:36.249044 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.249019 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef8297a_791a_4cfd_9880_3b6510c03642.slice/crio-88d883947759319f05f509264c5f1ff45b9e7c4af88121ba657bdd2347fcfce6 WatchSource:0}: Error finding container 88d883947759319f05f509264c5f1ff45b9e7c4af88121ba657bdd2347fcfce6: Status 404 returned error can't find the container with id 88d883947759319f05f509264c5f1ff45b9e7c4af88121ba657bdd2347fcfce6 Jul 14 19:46:36.267567 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.267545 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:36.274777 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.274752 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc720d8d6_c35c_437f_bcbd_fee227e73d46.slice/crio-6b79359d21631cf4afd691aafc5205e993b6dd0940ed5df50b61b36edd94702b WatchSource:0}: Error finding container 6b79359d21631cf4afd691aafc5205e993b6dd0940ed5df50b61b36edd94702b: Status 404 returned error can't find the container with id 6b79359d21631cf4afd691aafc5205e993b6dd0940ed5df50b61b36edd94702b Jul 14 19:46:36.284609 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.284592 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" Jul 14 19:46:36.288418 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.288397 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" Jul 14 19:46:36.291708 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.291686 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07428bc7_8a29_4650_b882_b2ba2fceca25.slice/crio-b4049eadf706c4ec42aebfc539b2150c0d91e42465cfdcb561efd32147cb169d WatchSource:0}: Error finding container b4049eadf706c4ec42aebfc539b2150c0d91e42465cfdcb561efd32147cb169d: Status 404 returned error can't find the container with id b4049eadf706c4ec42aebfc539b2150c0d91e42465cfdcb561efd32147cb169d Jul 14 19:46:36.296166 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.296146 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b529c87_9604_4046_bff6_a0a0a5a60154.slice/crio-c4267f9c68709151f710e540ef147121dc8bfc2830fa4742242680ffb063aa0d WatchSource:0}: Error finding container c4267f9c68709151f710e540ef147121dc8bfc2830fa4742242680ffb063aa0d: Status 404 returned error can't find the container with id c4267f9c68709151f710e540ef147121dc8bfc2830fa4742242680ffb063aa0d Jul 14 19:46:36.315091 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.315052 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njn4q" Jul 14 19:46:36.322695 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.322672 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb36332_2f1b_4aee_a15b_951633f6e6cc.slice/crio-b9126b7c1ec220244ed561057fdaf7c491368a6b6ab7b268fc06f49a9732a1ae WatchSource:0}: Error finding container b9126b7c1ec220244ed561057fdaf7c491368a6b6ab7b268fc06f49a9732a1ae: Status 404 returned error can't find the container with id b9126b7c1ec220244ed561057fdaf7c491368a6b6ab7b268fc06f49a9732a1ae Jul 14 19:46:36.333705 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.333679 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gnt9t" Jul 14 19:46:36.338512 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.338449 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:46:36.341054 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.341030 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa65ba4e_5b03_4aa4_be05_4d92cf63a8b8.slice/crio-0541332bcc91f6fb39f0cfd783cf8ea2fd738db9282fcba26b0b0c0f0d8df8a0 WatchSource:0}: Error finding container 0541332bcc91f6fb39f0cfd783cf8ea2fd738db9282fcba26b0b0c0f0d8df8a0: Status 404 returned error can't find the container with id 0541332bcc91f6fb39f0cfd783cf8ea2fd738db9282fcba26b0b0c0f0d8df8a0 Jul 14 19:46:36.345220 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.345201 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d713aa_4f27_416c_b5cd_55a61898b273.slice/crio-6d2606cd95f0e7d1c45d6c3a8954bffae220c9a8737a0dedd9f514cc7a73cfcd WatchSource:0}: Error finding container 6d2606cd95f0e7d1c45d6c3a8954bffae220c9a8737a0dedd9f514cc7a73cfcd: Status 404 returned error can't find the container with id 6d2606cd95f0e7d1c45d6c3a8954bffae220c9a8737a0dedd9f514cc7a73cfcd Jul 14 19:46:36.577108 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.577014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:36.577108 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.577057 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:36.577269 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.577155 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:36.577269 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.577159 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:36.577269 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.577209 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:46:37.577197288 +0000 UTC m=+3.025823711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:36.577269 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.577224 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:46:37.577217273 +0000 UTC m=+3.025843696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:36.659265 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:46:36.659221 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7429f60e293240c012ee36892b0b6e39.slice/crio-dbba705a069769f6d949b50ca6e889ab5479d173d9d8763add69b118dc483e49 WatchSource:0}: Error finding container dbba705a069769f6d949b50ca6e889ab5479d173d9d8763add69b118dc483e49: Status 404 returned error can't find the container with id dbba705a069769f6d949b50ca6e889ab5479d173d9d8763add69b118dc483e49 Jul 14 19:46:36.678307 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.678277 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:36.678541 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.678515 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:36.678641 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.678574 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:36.678641 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.678589 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:36.678729 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:36.678648 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:46:37.678630267 +0000 UTC m=+3.127256698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:36.918561 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.918445 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:41:35 +0000 UTC" deadline="2028-02-05 06:20:53.08971354 +0000 UTC" Jul 14 19:46:36.918561 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:36.918509 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13690h34m16.171208175s" Jul 14 19:46:37.032838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.032806 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:37.033005 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.032936 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:37.040143 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.040089 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"6d2606cd95f0e7d1c45d6c3a8954bffae220c9a8737a0dedd9f514cc7a73cfcd"} Jul 14 19:46:37.042137 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.042110 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gnt9t" event={"ID":"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8","Type":"ContainerStarted","Data":"0541332bcc91f6fb39f0cfd783cf8ea2fd738db9282fcba26b0b0c0f0d8df8a0"} Jul 14 19:46:37.043616 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.043593 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njn4q" event={"ID":"3cb36332-2f1b-4aee-a15b-951633f6e6cc","Type":"ContainerStarted","Data":"b9126b7c1ec220244ed561057fdaf7c491368a6b6ab7b268fc06f49a9732a1ae"} Jul 14 19:46:37.045848 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.045778 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" event={"ID":"8b529c87-9604-4046-bff6-a0a0a5a60154","Type":"ContainerStarted","Data":"c4267f9c68709151f710e540ef147121dc8bfc2830fa4742242680ffb063aa0d"} Jul 14 19:46:37.048966 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.048885 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerStarted","Data":"aa86e4deb1b0feed2fab5b8b3dbf4fe33ffae31e766abe7f63aad5ad8ebd86e6"} Jul 14 19:46:37.051838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.051755 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" event={"ID":"452d5b1c76cd4f857f719491897be84c","Type":"ContainerStarted","Data":"4541d2f38ebdc11697405a06ddf34a33d33df11cddfef2e36145f595cee887f6"} Jul 14 19:46:37.054408 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.054383 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" event={"ID":"07428bc7-8a29-4650-b882-b2ba2fceca25","Type":"ContainerStarted","Data":"b4049eadf706c4ec42aebfc539b2150c0d91e42465cfdcb561efd32147cb169d"} Jul 14 19:46:37.061364 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.061340 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2zjw5" event={"ID":"c720d8d6-c35c-437f-bcbd-fee227e73d46","Type":"ContainerStarted","Data":"6b79359d21631cf4afd691aafc5205e993b6dd0940ed5df50b61b36edd94702b"} Jul 14 19:46:37.062663 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.062611 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qqgd9" event={"ID":"fef8297a-791a-4cfd-9880-3b6510c03642","Type":"ContainerStarted","Data":"88d883947759319f05f509264c5f1ff45b9e7c4af88121ba657bdd2347fcfce6"} Jul 14 19:46:37.064874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.064850 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fxc4r" event={"ID":"53011eae-e1ec-4f43-aee9-aa5ac3a52859","Type":"ContainerStarted","Data":"21003970f75cb3a1ee8bfd9bba2a329017b354e696e5c0d498dab6e05b0fc4e7"} Jul 14 19:46:37.067317 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.067294 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" event={"ID":"7429f60e293240c012ee36892b0b6e39","Type":"ContainerStarted","Data":"dbba705a069769f6d949b50ca6e889ab5479d173d9d8763add69b118dc483e49"} Jul 14 19:46:37.187579 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.187492 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:37.585794 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.585715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:37.585794 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.585769 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:37.586009 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.585915 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:37.586009 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.585978 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:46:39.585959202 +0000 UTC m=+5.034585630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:37.586379 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.586363 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:37.586465 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.586414 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:46:39.586399206 +0000 UTC m=+5.035025635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:37.686421 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.686386 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:37.686632 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.686557 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:37.686632 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.686577 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:37.686632 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.686590 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:37.686817 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:37.686649 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:46:39.68663158 +0000 UTC m=+5.135258026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:37.919410 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.919329 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:41:35 +0000 UTC" deadline="2028-04-08 22:35:54.376632514 +0000 UTC" Jul 14 19:46:37.919410 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:37.919369 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15218h49m16.457267698s" Jul 14 19:46:38.031181 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:38.031150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:38.031367 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:38.031280 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:38.031693 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:38.031667 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:38.031796 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:38.031776 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:38.910761 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:38.910467 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:39.031736 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:39.031702 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:39.032369 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.031850 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:39.604238 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:39.604200 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:39.604428 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:39.604256 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:39.604428 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.604347 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:39.604428 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.604364 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:39.604428 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.604417 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:46:43.604397067 +0000 UTC m=+9.053023493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:39.604632 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.604435 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:46:43.604426908 +0000 UTC m=+9.053053337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:39.704711 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:39.704676 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:39.704957 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.704905 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:39.704957 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.704932 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:39.704957 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.704946 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:39.705273 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:39.705008 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:46:43.704988501 +0000 UTC m=+9.153614937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:40.031747 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:40.031085 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:40.031747 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:40.031200 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:40.031747 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:40.031548 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:40.031747 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:40.031656 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:41.030954 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:41.030465 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:41.030954 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:41.030599 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:42.031036 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:42.030988 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:42.031533 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:42.030997 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:42.031533 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:42.031145 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:42.031533 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:42.031197 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:43.030927 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:43.030751 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:43.030927 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.030898 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:43.637760 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:43.637713 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:43.637951 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:43.637773 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:43.638023 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.637950 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:43.638170 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.638022 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:46:51.638002653 +0000 UTC m=+17.086629109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:43.638261 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.638216 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:43.638315 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.638283 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:46:51.638265326 +0000 UTC m=+17.086891749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:43.738500 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:43.738328 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:43.738500 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.738492 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:43.738733 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.738513 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:43.738733 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.738525 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:43.738733 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:43.738589 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:46:51.738572138 +0000 UTC m=+17.187198579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:44.030508 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:44.030430 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:44.030508 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:44.030450 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:44.030711 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:44.030567 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:44.030711 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:44.030653 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:45.031631 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:45.031426 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:45.031631 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:45.031564 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:46.031118 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:46.031055 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:46.031118 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:46.031130 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:46.031340 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:46.031218 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:46.031399 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:46.031362 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:47.030553 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:47.030516 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:47.031008 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:47.030662 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:48.030603 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:48.030549 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:48.031048 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:48.030549 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:48.031048 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:48.030683 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:48.031048 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:48.030758 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:49.030332 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:49.030300 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:49.030532 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:49.030439 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:50.030651 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:50.030612 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:50.031089 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:50.030749 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:50.031089 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:50.030804 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:50.031089 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:50.030909 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:51.030919 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:51.030882 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:51.031370 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.031022 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:51.698653 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:51.698614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:51.698653 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:51.698659 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:51.698886 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.698780 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:51.698886 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.698842 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:47:07.698823667 +0000 UTC m=+33.147450095 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:51.698886 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.698858 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:51.699007 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.698903 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:47:07.698887837 +0000 UTC m=+33.147514264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:51.799448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:51.799400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:51.799653 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.799594 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:51.799653 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.799622 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:51.799653 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.799635 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:51.799812 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:51.799698 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:47:07.799676849 +0000 UTC m=+33.248303288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:52.030715 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:52.030637 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:52.030715 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:52.030672 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:52.030902 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:52.030747 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:52.030902 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:52.030884 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:53.031039 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:53.031006 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:53.031496 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:53.031171 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:54.030954 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:54.030919 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:54.030954 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:54.030946 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:54.031398 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:54.031087 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:54.031398 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:54.031166 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:55.031187 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:55.031150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:55.031612 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:55.031265 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:56.030792 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.030421 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:56.030894 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.030448 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:56.030928 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:56.030910 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:56.030958 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:56.030942 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:56.109698 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.109662 2553 generic.go:356] "Generic (PLEG): container finished" podID="7429f60e293240c012ee36892b0b6e39" containerID="2fe770047e79faa53eb4153374457c35d77d6095daf4c7bb0c59e28cbe9437f4" exitCode=0 Jul 14 19:46:56.110589 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.109760 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" event={"ID":"7429f60e293240c012ee36892b0b6e39","Type":"ContainerDied","Data":"2fe770047e79faa53eb4153374457c35d77d6095daf4c7bb0c59e28cbe9437f4"} Jul 14 19:46:56.113457 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.113432 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:46:56.113899 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.113863 2553 generic.go:356] "Generic (PLEG): container finished" podID="37d713aa-4f27-416c-b5cd-55a61898b273" containerID="2f076fc5fcbaf32ce133ac787e5db5bfc21dffa1b239ad7cc544a26a07596f81" exitCode=1 Jul 14 19:46:56.114010 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.113950 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"70756280afe2b79519bac676078131a4bf695b34a1c68cd278b8b8c0018dcb6a"} Jul 14 19:46:56.114107 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.114022 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"24c9750d878c29d54d20a92462dc83de00d0116b1c9e9b5cff2ded98fa2a37fd"} Jul 14 19:46:56.114107 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.114087 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"e2d4a5d15d246496b6c0e8fdddbe5b994d3ca0f64831d1188e1b7a2317f29548"} Jul 14 19:46:56.114201 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.114131 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"8a2c17a6dafbe606754b15b9f1221211d37a2842f4e7bd1380f35d602f084e97"} Jul 14 19:46:56.114201 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.114175 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerDied","Data":"2f076fc5fcbaf32ce133ac787e5db5bfc21dffa1b239ad7cc544a26a07596f81"} Jul 14 19:46:56.114286 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.114224 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"59e03d2702a276b9eba78c8c95d22163c6d499c3c1a83a3e17dee6831f34807f"} Jul 14 19:46:56.115904 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.115710 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gnt9t" event={"ID":"fa65ba4e-5b03-4aa4-be05-4d92cf63a8b8","Type":"ContainerStarted","Data":"288d60d9e6e050e2f7b9fda04c01da6636093d5f7ab689b090c480cd46ad54af"} Jul 14 19:46:56.117478 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.117453 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njn4q" event={"ID":"3cb36332-2f1b-4aee-a15b-951633f6e6cc","Type":"ContainerStarted","Data":"ae1a5bd99a2c82f4843429ac22b719167ec96a969223bc2c3b05055dd91f34ee"} Jul 14 19:46:56.121801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.121737 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" event={"ID":"8b529c87-9604-4046-bff6-a0a0a5a60154","Type":"ContainerStarted","Data":"62345029cdb7dc65b3691ff7b61e0902862d82473953bcc292ae4bbe83f7fc14"} Jul 14 19:46:56.123632 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.123609 2553 generic.go:356] "Generic (PLEG): container finished" podID="9449ff7f-e1fe-4a43-a747-1091a98dd62c" containerID="4ffb3c9730fccc845eb25fe1eb523d08a1bb31fe9f79fe9aa8969d31a7890c1a" exitCode=0 Jul 14 19:46:56.123881 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.123860 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerDied","Data":"4ffb3c9730fccc845eb25fe1eb523d08a1bb31fe9f79fe9aa8969d31a7890c1a"} Jul 14 19:46:56.125995 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.125945 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" event={"ID":"452d5b1c76cd4f857f719491897be84c","Type":"ContainerStarted","Data":"7cc903df32d99fe6b8fcf14a719f9625f9edc2dff17f56db1a0560c5504febab"} Jul 14 19:46:56.127544 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.127521 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" event={"ID":"07428bc7-8a29-4650-b882-b2ba2fceca25","Type":"ContainerStarted","Data":"f3c94da69639f63323b620365ab421fbaeb88056ec83e61cdd8217ea82869589"} Jul 14 19:46:56.128937 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.128882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2zjw5" event={"ID":"c720d8d6-c35c-437f-bcbd-fee227e73d46","Type":"ContainerStarted","Data":"ff3cf2df729bce22586ea859019ff3fd6ca0063550ce7b93724d4e2a2431a1cb"} Jul 14 19:46:56.130373 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.130350 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fxc4r" event={"ID":"53011eae-e1ec-4f43-aee9-aa5ac3a52859","Type":"ContainerStarted","Data":"11f4baf1016c5f76d05312468ebc780616d99b6297c37a97d8c3e6174c7f82fe"} Jul 14 19:46:56.145933 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.145884 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gnt9t" podStartSLOduration=2.1225020629999998 podStartE2EDuration="21.145867587s" totalImagesPullingTime="19.023365524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:56.145440451 +0000 UTC m=+21.594066896" watchObservedRunningTime="2026-07-14 19:46:56.145867587 +0000 UTC m=+21.594494036" Jul 14 19:46:56.162882 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.162837 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-njn4q" podStartSLOduration=2.262079065 podStartE2EDuration="21.162821165s" totalImagesPullingTime="18.9007421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:56.1622336 +0000 UTC m=+21.610860057" watchObservedRunningTime="2026-07-14 19:46:56.162821165 +0000 UTC m=+21.611447609" Jul 14 19:46:56.233727 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.233676 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dbt6j" podStartSLOduration=2.306318644 podStartE2EDuration="21.233661679s" totalImagesPullingTime="18.927343035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:56.208206456 +0000 UTC m=+21.656832899" watchObservedRunningTime="2026-07-14 19:46:56.233661679 +0000 UTC m=+21.682288124" Jul 14 19:46:56.263770 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.263719 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2zjw5" podStartSLOduration=7.68227457 podStartE2EDuration="21.263703375s" totalImagesPullingTime="13.581428805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:56.234152786 +0000 UTC m=+21.682779235" watchObservedRunningTime="2026-07-14 19:46:56.263703375 +0000 UTC m=+21.712329821" Jul 14 19:46:56.286432 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.286385 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" podStartSLOduration=21.286368469 podStartE2EDuration="21.286368469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:56.264038546 +0000 UTC m=+21.712665004" watchObservedRunningTime="2026-07-14 19:46:56.286368469 +0000 UTC m=+21.734994913" Jul 14 19:46:56.286585 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.286449 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fxc4r" podStartSLOduration=2.226504262 podStartE2EDuration="21.28644466s" totalImagesPullingTime="19.059940398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:56.286111993 +0000 UTC m=+21.734738438" watchObservedRunningTime="2026-07-14 19:46:56.28644466 +0000 UTC m=+21.735071155" Jul 14 19:46:56.396811 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.396721 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:56.860897 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.860875 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 19:46:56.938010 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.937869 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:46:56.860891496Z","UUID":"d0128881-450f-475f-aa64-b3392a6581f0","Handler":null,"Name":"","Endpoint":""} Jul 14 19:46:56.939353 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.939336 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 19:46:56.939454 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:56.939372 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 19:46:57.005238 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.005203 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:57.031192 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.031159 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:57.031340 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:57.031281 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:57.105703 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.105671 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:57.133565 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.133534 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" event={"ID":"7429f60e293240c012ee36892b0b6e39","Type":"ContainerStarted","Data":"cdaeb9733434c562d757b6f45e916d13e69f3a50e7177072096dd48d97b6344f"} Jul 14 19:46:57.135139 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.135118 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" event={"ID":"07428bc7-8a29-4650-b882-b2ba2fceca25","Type":"ContainerStarted","Data":"949f5898e57d8688bda7b72d4ed950f82671e20e38f2819946eb37176ae7fff7"} Jul 14 19:46:57.136378 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.136358 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qqgd9" event={"ID":"fef8297a-791a-4cfd-9880-3b6510c03642","Type":"ContainerStarted","Data":"d0bdff32f13d25bce130b9e63dce92b48d8eb89552cb9951f371af8b003c6555"} Jul 14 19:46:57.152259 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.152222 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" podStartSLOduration=22.152208669 podStartE2EDuration="22.152208669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:57.151932495 +0000 UTC m=+22.600558939" watchObservedRunningTime="2026-07-14 19:46:57.152208669 +0000 UTC m=+22.600835113" Jul 14 19:46:57.168316 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.168274 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qqgd9" podStartSLOduration=3.129326912 podStartE2EDuration="22.168261322s" totalImagesPullingTime="19.03893441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:57.168133989 +0000 UTC m=+22.616760445" watchObservedRunningTime="2026-07-14 19:46:57.168261322 +0000 UTC m=+22.616887782" Jul 14 19:46:57.176442 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.176421 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:57.523590 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.523560 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:57.721515 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.721486 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:57.883345 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.883270 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:57.883864 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:57.883850 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:58.030629 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:58.030607 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:46:58.030785 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:58.030607 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:46:58.030785 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:58.030694 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:46:58.030785 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:58.030767 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:46:58.155921 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:58.155834 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:58.156316 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:58.156304 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2zjw5" Jul 14 19:46:58.296545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:58.296306 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:59.031077 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.031032 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:46:59.031272 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:46:59.031174 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:46:59.144181 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.144154 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:46:59.144637 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.144609 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"1e8e69b85ce00e131a4273f0f8b5f6f51bc52e8d791e7d629f9f4c5af7e21df2"} Jul 14 19:46:59.513679 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.513589 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:59.613545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.613504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:59.714113 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.714077 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:59.891681 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:46:59.891600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:00.030576 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:00.030539 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:00.030764 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:00.030539 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:00.030764 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:00.030667 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:00.030764 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:00.030712 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:00.148576 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:00.148500 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" event={"ID":"07428bc7-8a29-4650-b882-b2ba2fceca25","Type":"ContainerStarted","Data":"3efb1abcaebbb6d0724ae6ae179c96c63bd1781a6494f8785acb5a5ffe493457"} Jul 14 19:47:00.169453 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:00.169392 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7frp" podStartSLOduration=2.859183699 podStartE2EDuration="25.169370938s" totalImagesPullingTime="22.310187239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:00.169103314 +0000 UTC m=+25.617729758" watchObservedRunningTime="2026-07-14 19:47:00.169370938 +0000 UTC m=+25.617997383" Jul 14 19:47:00.207740 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:00.207705 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:00.276741 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:00.276714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:01.030726 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:01.030691 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:01.031329 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:01.030818 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:47:01.370657 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:01.370585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:01.469722 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:01.469681 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:01.606209 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:01.606179 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:01.674219 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:01.674146 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:02.030780 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.030487 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:02.031287 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.030487 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:02.031287 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:02.030851 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:02.031287 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:02.030903 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:02.156440 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.156418 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:47:02.156791 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.156768 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"4fc60798f36e67512b942fd08eaa3ce5cbeb0d502fe65b38cd68a70ba7f43167"} Jul 14 19:47:02.157116 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.157047 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:47:02.157210 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.157175 2553 scope.go:117] "RemoveContainer" containerID="2f076fc5fcbaf32ce133ac787e5db5bfc21dffa1b239ad7cc544a26a07596f81" Jul 14 19:47:02.158845 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.158819 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerStarted","Data":"c1acf3cf5ffe80d294b625d841633c37b38c1860b037b686b802b90f1a7f6ecd"} Jul 14 19:47:02.173572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.173550 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:47:02.692864 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:02.692778 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:03.030697 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.030624 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:03.030844 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:03.030730 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:47:03.164016 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.163990 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:47:03.164419 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.164385 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" event={"ID":"37d713aa-4f27-416c-b5cd-55a61898b273","Type":"ContainerStarted","Data":"0dea6583fb2fd27e644ce57398d6c3cfff6e05171d86d02524492452aacee859"} Jul 14 19:47:03.164535 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.164454 2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 19:47:03.167853 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.164992 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:47:03.169403 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.169373 2553 generic.go:356] "Generic (PLEG): container finished" podID="9449ff7f-e1fe-4a43-a747-1091a98dd62c" containerID="c1acf3cf5ffe80d294b625d841633c37b38c1860b037b686b802b90f1a7f6ecd" exitCode=0 Jul 14 19:47:03.169493 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.169412 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerDied","Data":"c1acf3cf5ffe80d294b625d841633c37b38c1860b037b686b802b90f1a7f6ecd"} Jul 14 19:47:03.180921 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.180901 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:47:03.192473 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.192428 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" podStartSLOduration=9.234956876 podStartE2EDuration="28.192416016s" totalImagesPullingTime="18.95745914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:03.192135963 +0000 UTC m=+28.640762407" watchObservedRunningTime="2026-07-14 19:47:03.192416016 +0000 UTC m=+28.641042495" Jul 14 19:47:03.446876 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:03.446787 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:47:04.030556 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:04.030520 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:04.030707 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:04.030520 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:04.030707 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:04.030632 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:04.030707 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:04.030680 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:05.031849 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:05.031826 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:05.032228 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:05.031905 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:47:05.175733 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:05.175695 2553 generic.go:356] "Generic (PLEG): container finished" podID="9449ff7f-e1fe-4a43-a747-1091a98dd62c" containerID="45ee58760bc744c7c08b8dcb536bd972366eb8b61100c1ab7831ad0cbab02835" exitCode=0 Jul 14 19:47:05.175882 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:05.175780 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerDied","Data":"45ee58760bc744c7c08b8dcb536bd972366eb8b61100c1ab7831ad0cbab02835"} Jul 14 19:47:06.030838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:06.030657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:06.030980 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:06.030657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:06.030980 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:06.030908 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:06.030980 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:06.030968 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:06.644574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:06.644485 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:06.780696 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:06.780666 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:06.902261 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:06.902177 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:07.031280 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.031246 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:07.031455 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.031377 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:47:07.049483 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.049445 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:07.181970 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.181884 2553 generic.go:356] "Generic (PLEG): container finished" podID="9449ff7f-e1fe-4a43-a747-1091a98dd62c" containerID="b3b84d5146da1328f35ae43116d08a2ba47d07e27129c8866fe30e6db33d25f0" exitCode=0 Jul 14 19:47:07.181970 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.181924 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerDied","Data":"b3b84d5146da1328f35ae43116d08a2ba47d07e27129c8866fe30e6db33d25f0"} Jul 14 19:47:07.221314 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.221284 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:07.719697 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.719662 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:07.720113 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.719723 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:07.720113 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.719805 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:07.720113 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.719826 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:07.720113 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.719856 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret podName:437754cb-a459-4410-b75f-8ee2f10721ba nodeName:}" failed. No retries permitted until 2026-07-14 19:47:39.719842961 +0000 UTC m=+65.168469384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret") pod "global-pull-secret-syncer-26pgr" (UID: "437754cb-a459-4410-b75f-8ee2f10721ba") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:07.720113 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.719903 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs podName:ebae7f13-3078-42b4-b221-aa64593e0bfb nodeName:}" failed. No retries permitted until 2026-07-14 19:47:39.719885116 +0000 UTC m=+65.168511542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs") pod "network-metrics-daemon-hj4v9" (UID: "ebae7f13-3078-42b4-b221-aa64593e0bfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:07.820640 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:07.820607 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:07.820815 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.820770 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:07.820815 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.820790 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:07.820815 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.820801 2553 projected.go:196] Error preparing data for projected volume kube-api-access-trllt for pod openshift-network-diagnostics/network-check-target-8fbqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:07.820907 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:07.820862 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt podName:197f9801-26a1-4614-9ef8-9e89a67bdb1d nodeName:}" failed. No retries permitted until 2026-07-14 19:47:39.820843137 +0000 UTC m=+65.269469561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-trllt" (UniqueName: "kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt") pod "network-check-target-8fbqx" (UID: "197f9801-26a1-4614-9ef8-9e89a67bdb1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:08.030520 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.030483 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:08.030520 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.030515 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:08.030686 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:08.030617 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:08.030758 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:08.030737 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:08.104274 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.104245 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8fbqx"] Jul 14 19:47:08.108163 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.108131 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-26pgr"] Jul 14 19:47:08.108639 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.108620 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hj4v9"] Jul 14 19:47:08.108744 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.108730 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:08.108855 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:08.108831 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:47:08.184662 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.184635 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:08.184823 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:08.184636 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:08.184823 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:08.184744 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:08.184823 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:08.184812 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:10.030370 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:10.030337 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:10.031127 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:10.030343 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:10.031127 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:10.030471 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hj4v9" podUID="ebae7f13-3078-42b4-b221-aa64593e0bfb" Jul 14 19:47:10.031127 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:10.030348 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:10.031127 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:10.030541 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8fbqx" podUID="197f9801-26a1-4614-9ef8-9e89a67bdb1d" Jul 14 19:47:10.031127 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:10.030633 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26pgr" podUID="437754cb-a459-4410-b75f-8ee2f10721ba" Jul 14 19:47:11.353350 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.353300 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:11.762286 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.762198 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:11.940409 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.940229 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeReady" Jul 14 19:47:11.940555 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.940539 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 19:47:11.974546 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.974420 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8f854fddd-dtm2f"] Jul 14 19:47:11.985862 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.985833 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:11.986161 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.986135 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-99jl6"] Jul 14 19:47:11.988366 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.988342 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 19:47:11.989006 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.988984 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 19:47:11.989112 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.989023 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4vjs\"" Jul 14 19:47:11.989219 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.989188 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 19:47:11.994582 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.994560 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 19:47:11.996056 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.996036 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k9zs2"] Jul 14 19:47:11.996209 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.996176 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:11.998553 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.998518 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 19:47:11.998687 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.998565 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46xb9\"" Jul 14 19:47:11.998749 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.998718 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 19:47:11.998804 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:11.998748 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 19:47:12.008464 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.008427 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f854fddd-dtm2f"] Jul 14 19:47:12.008464 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.008464 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-99jl6"] Jul 14 19:47:12.008642 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.008496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k9zs2"] Jul 14 19:47:12.008642 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.008598 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.011032 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.010989 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 19:47:12.011032 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.011019 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 19:47:12.011032 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.011024 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 19:47:12.011265 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.011051 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 19:47:12.011359 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.011346 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpj6w\"" Jul 14 19:47:12.030638 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.030548 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:12.030638 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.030624 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:12.030771 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.030724 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:12.032959 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.032946 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:47:12.033020 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.032981 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:47:12.033091 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.033049 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xm9xl\"" Jul 14 19:47:12.033179 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.033165 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:47:12.033245 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.033230 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvlkh\"" Jul 14 19:47:12.033291 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.033248 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:47:12.086150 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.086127 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pcsfp"] Jul 14 19:47:12.091535 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.091516 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.093937 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.093922 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 19:47:12.094037 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.094024 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 19:47:12.094102 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.094094 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7w2mk\"" Jul 14 19:47:12.098844 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.098783 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcsfp"] Jul 14 19:47:12.154788 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154757 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8249e15-4445-4b01-a11e-fe7a705b62b3-image-registry-private-configuration\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.154956 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154794 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f83805c3-112a-441e-a06b-dd01f9a1f2b1-cert\") pod \"ingress-canary-99jl6\" (UID: \"f83805c3-112a-441e-a06b-dd01f9a1f2b1\") " pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.154956 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d35a5a8b-76ca-4120-9695-8e090145d939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.154956 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154847 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8249e15-4445-4b01-a11e-fe7a705b62b3-ca-trust-extracted\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.154956 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154901 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d35a5a8b-76ca-4120-9695-8e090145d939-data-volume\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.154956 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154941 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8249e15-4445-4b01-a11e-fe7a705b62b3-trusted-ca\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.155204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.154971 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8249e15-4445-4b01-a11e-fe7a705b62b3-installation-pull-secrets\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.155204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155020 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxcl\" (UniqueName: \"kubernetes.io/projected/f83805c3-112a-441e-a06b-dd01f9a1f2b1-kube-api-access-zcxcl\") pod \"ingress-canary-99jl6\" (UID: \"f83805c3-112a-441e-a06b-dd01f9a1f2b1\") " pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.155204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155119 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d35a5a8b-76ca-4120-9695-8e090145d939-crio-socket\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.155204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d35a5a8b-76ca-4120-9695-8e090145d939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.155204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155159 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8249e15-4445-4b01-a11e-fe7a705b62b3-registry-certificates\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.155204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155174 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-bound-sa-token\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.155457 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155238 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnch\" (UniqueName: \"kubernetes.io/projected/d35a5a8b-76ca-4120-9695-8e090145d939-kube-api-access-7hnch\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.155457 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155260 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-registry-tls\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.155457 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.155275 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnz22\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-kube-api-access-hnz22\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.193843 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.193807 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:12.196971 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.196946 2553 generic.go:356] "Generic (PLEG): container finished" podID="9449ff7f-e1fe-4a43-a747-1091a98dd62c" containerID="74ae55f9869e79d043df2cbef111f6e0b2b2e1b6b280bbc2910afd0db2bc248a" exitCode=0 Jul 14 19:47:12.197122 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.196981 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerDied","Data":"74ae55f9869e79d043df2cbef111f6e0b2b2e1b6b280bbc2910afd0db2bc248a"} Jul 14 19:47:12.256343 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256319 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-bound-sa-token\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.256408 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256388 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnch\" (UniqueName: \"kubernetes.io/projected/d35a5a8b-76ca-4120-9695-8e090145d939-kube-api-access-7hnch\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.256448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256411 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-registry-tls\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.256448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256436 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnz22\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-kube-api-access-hnz22\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.256549 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256480 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-metrics-tls\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.256634 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256616 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-config-volume\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.256696 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkh59\" (UniqueName: \"kubernetes.io/projected/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-kube-api-access-jkh59\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.256750 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256731 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8249e15-4445-4b01-a11e-fe7a705b62b3-image-registry-private-configuration\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.256792 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256771 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f83805c3-112a-441e-a06b-dd01f9a1f2b1-cert\") pod \"ingress-canary-99jl6\" (UID: \"f83805c3-112a-441e-a06b-dd01f9a1f2b1\") " pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.256874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256790 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d35a5a8b-76ca-4120-9695-8e090145d939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.256874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8249e15-4445-4b01-a11e-fe7a705b62b3-ca-trust-extracted\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.256874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256835 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d35a5a8b-76ca-4120-9695-8e090145d939-data-volume\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.256874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256852 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8249e15-4445-4b01-a11e-fe7a705b62b3-trusted-ca\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.256874 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256871 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8249e15-4445-4b01-a11e-fe7a705b62b3-installation-pull-secrets\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.257141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256905 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxcl\" (UniqueName: \"kubernetes.io/projected/f83805c3-112a-441e-a06b-dd01f9a1f2b1-kube-api-access-zcxcl\") pod \"ingress-canary-99jl6\" (UID: \"f83805c3-112a-441e-a06b-dd01f9a1f2b1\") " pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.257141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256929 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-tmp-dir\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.257141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256966 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d35a5a8b-76ca-4120-9695-8e090145d939-crio-socket\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.257141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.256994 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d35a5a8b-76ca-4120-9695-8e090145d939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.257141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.257019 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8249e15-4445-4b01-a11e-fe7a705b62b3-registry-certificates\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.257464 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.257441 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d35a5a8b-76ca-4120-9695-8e090145d939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.258034 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.257719 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8249e15-4445-4b01-a11e-fe7a705b62b3-ca-trust-extracted\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.258034 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.257745 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d35a5a8b-76ca-4120-9695-8e090145d939-data-volume\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.258034 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.257928 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d35a5a8b-76ca-4120-9695-8e090145d939-crio-socket\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.258034 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.258002 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8249e15-4445-4b01-a11e-fe7a705b62b3-trusted-ca\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.258305 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.258257 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8249e15-4445-4b01-a11e-fe7a705b62b3-registry-certificates\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.261469 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.261447 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8249e15-4445-4b01-a11e-fe7a705b62b3-installation-pull-secrets\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.261469 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.261463 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8249e15-4445-4b01-a11e-fe7a705b62b3-image-registry-private-configuration\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.261625 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.261601 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d35a5a8b-76ca-4120-9695-8e090145d939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.261694 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.261676 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f83805c3-112a-441e-a06b-dd01f9a1f2b1-cert\") pod \"ingress-canary-99jl6\" (UID: \"f83805c3-112a-441e-a06b-dd01f9a1f2b1\") " pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.262014 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.261994 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-registry-tls\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.268127 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.268102 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnz22\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-kube-api-access-hnz22\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.268273 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.268255 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxcl\" (UniqueName: \"kubernetes.io/projected/f83805c3-112a-441e-a06b-dd01f9a1f2b1-kube-api-access-zcxcl\") pod \"ingress-canary-99jl6\" (UID: \"f83805c3-112a-441e-a06b-dd01f9a1f2b1\") " pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.269057 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.269027 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8249e15-4445-4b01-a11e-fe7a705b62b3-bound-sa-token\") pod \"image-registry-8f854fddd-dtm2f\" (UID: \"a8249e15-4445-4b01-a11e-fe7a705b62b3\") " pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.269291 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.269271 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnch\" (UniqueName: \"kubernetes.io/projected/d35a5a8b-76ca-4120-9695-8e090145d939-kube-api-access-7hnch\") pod \"insights-runtime-extractor-k9zs2\" (UID: \"d35a5a8b-76ca-4120-9695-8e090145d939\") " pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.336202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.336178 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:12.347277 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.347253 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-99jl6" Jul 14 19:47:12.353622 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.353601 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k9zs2" Jul 14 19:47:12.357904 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.357878 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-tmp-dir\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.358199 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.358148 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-metrics-tls\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.358199 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.358180 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-config-volume\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.358416 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.358202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkh59\" (UniqueName: \"kubernetes.io/projected/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-kube-api-access-jkh59\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.358416 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.358303 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-tmp-dir\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.358912 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.358894 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-config-volume\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.360785 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.360764 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-metrics-tls\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.367235 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.367211 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkh59\" (UniqueName: \"kubernetes.io/projected/995ad6e1-3b0a-462d-83b0-af9e5a1178fc-kube-api-access-jkh59\") pod \"dns-default-pcsfp\" (UID: \"995ad6e1-3b0a-462d-83b0-af9e5a1178fc\") " pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.400781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.400756 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:12.469022 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.468983 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:12.539379 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.539310 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:12.546238 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.546154 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-99jl6"] Jul 14 19:47:12.550506 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:12.550479 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83805c3_112a_441e_a06b_dd01f9a1f2b1.slice/crio-351b9db64f6b4df6c355c739be497c0f54bc13f07b8a4931fe85a2fa57418056 WatchSource:0}: Error finding container 351b9db64f6b4df6c355c739be497c0f54bc13f07b8a4931fe85a2fa57418056: Status 404 returned error can't find the container with id 351b9db64f6b4df6c355c739be497c0f54bc13f07b8a4931fe85a2fa57418056 Jul 14 19:47:12.557495 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.557470 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f854fddd-dtm2f"] Jul 14 19:47:12.560745 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.560712 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k9zs2"] Jul 14 19:47:12.561806 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:12.561780 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8249e15_4445_4b01_a11e_fe7a705b62b3.slice/crio-ac6cb583c6a30ae14e54e4cefe5d28b36359868d769baf10043657930bc11c2d WatchSource:0}: Error finding container ac6cb583c6a30ae14e54e4cefe5d28b36359868d769baf10043657930bc11c2d: Status 404 returned error can't find the container with id ac6cb583c6a30ae14e54e4cefe5d28b36359868d769baf10043657930bc11c2d Jul 14 19:47:12.563980 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:12.563960 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35a5a8b_76ca_4120_9695_8e090145d939.slice/crio-344658e1acdefaa16e2245fa6774adea14c2f0704d4e9f2bbb77b78b39534ff9 WatchSource:0}: Error finding container 344658e1acdefaa16e2245fa6774adea14c2f0704d4e9f2bbb77b78b39534ff9: Status 404 returned error can't find the container with id 344658e1acdefaa16e2245fa6774adea14c2f0704d4e9f2bbb77b78b39534ff9 Jul 14 19:47:12.575119 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.575096 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcsfp"] Jul 14 19:47:12.579461 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:12.579436 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995ad6e1_3b0a_462d_83b0_af9e5a1178fc.slice/crio-75df1b7dfe3c6a04941c73fbc774e3a3fb1f038a63756b33f0cc64e06eadf0fe WatchSource:0}: Error finding container 75df1b7dfe3c6a04941c73fbc774e3a3fb1f038a63756b33f0cc64e06eadf0fe: Status 404 returned error can't find the container with id 75df1b7dfe3c6a04941c73fbc774e3a3fb1f038a63756b33f0cc64e06eadf0fe Jul 14 19:47:12.638878 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.638720 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:12.741091 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:12.741053 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:13.210209 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.209272 2553 generic.go:356] "Generic (PLEG): container finished" podID="9449ff7f-e1fe-4a43-a747-1091a98dd62c" containerID="dcc3b5cda237dbde1d0de6523323a17d859dee174cf895caf9c36c53b5903276" exitCode=0 Jul 14 19:47:13.210209 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.209351 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerDied","Data":"dcc3b5cda237dbde1d0de6523323a17d859dee174cf895caf9c36c53b5903276"} Jul 14 19:47:13.218308 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.215687 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-99jl6" event={"ID":"f83805c3-112a-441e-a06b-dd01f9a1f2b1","Type":"ContainerStarted","Data":"351b9db64f6b4df6c355c739be497c0f54bc13f07b8a4931fe85a2fa57418056"} Jul 14 19:47:13.221521 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.221430 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9zs2" event={"ID":"d35a5a8b-76ca-4120-9695-8e090145d939","Type":"ContainerStarted","Data":"344658e1acdefaa16e2245fa6774adea14c2f0704d4e9f2bbb77b78b39534ff9"} Jul 14 19:47:13.227300 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.227194 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" event={"ID":"a8249e15-4445-4b01-a11e-fe7a705b62b3","Type":"ContainerStarted","Data":"4a40c5eb603129eb6b3137c10c31b24fde620c1e349631c3d601302c61f99127"} Jul 14 19:47:13.227300 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.227248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" event={"ID":"a8249e15-4445-4b01-a11e-fe7a705b62b3","Type":"ContainerStarted","Data":"ac6cb583c6a30ae14e54e4cefe5d28b36359868d769baf10043657930bc11c2d"} Jul 14 19:47:13.231569 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.231284 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:13.232298 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.232219 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcsfp" event={"ID":"995ad6e1-3b0a-462d-83b0-af9e5a1178fc","Type":"ContainerStarted","Data":"75df1b7dfe3c6a04941c73fbc774e3a3fb1f038a63756b33f0cc64e06eadf0fe"} Jul 14 19:47:13.856618 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.856488 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:13.956817 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:13.956752 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:14.239892 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:14.239481 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" event={"ID":"9449ff7f-e1fe-4a43-a747-1091a98dd62c","Type":"ContainerStarted","Data":"c2de1d0d5fd3bff9ea3a62c017258365bb61e8004027d765496ea446c44a6f87"} Jul 14 19:47:14.264558 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:14.264512 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hp9tz" podStartSLOduration=12.897832831 podStartE2EDuration="39.264491494s" totalImagesPullingTime="26.366658663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:14.263748595 +0000 UTC m=+39.712375039" watchObservedRunningTime="2026-07-14 19:47:14.264491494 +0000 UTC m=+39.713117940" Jul 14 19:47:14.265416 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:14.265373 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" podStartSLOduration=8.265360214 podStartE2EDuration="8.265360214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:13.258581985 +0000 UTC m=+38.707208431" watchObservedRunningTime="2026-07-14 19:47:14.265360214 +0000 UTC m=+39.713986658" Jul 14 19:47:14.391912 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:14.391878 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:14.459498 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:14.459458 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:15.112105 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:15.112073 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:15.237467 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:15.237424 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:15.339145 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:15.339110 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:15.458353 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:15.458317 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:15.549941 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:15.549903 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:16.285877 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:16.285845 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:16.542800 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:16.542729 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:16.646591 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:16.646553 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:16.746951 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:16.746920 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:18.016679 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:18.016558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:18.117637 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:18.117608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:19.254025 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.253934 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcsfp" event={"ID":"995ad6e1-3b0a-462d-83b0-af9e5a1178fc","Type":"ContainerStarted","Data":"602ee9ba5bacf1438697be29a5e68b27ed2c4f11ce431203848568ab3fbf4e9d"} Jul 14 19:47:19.254025 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.253986 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcsfp" event={"ID":"995ad6e1-3b0a-462d-83b0-af9e5a1178fc","Type":"ContainerStarted","Data":"37df4f464dcce05411bf03ddd7b0d065a8040514b4393aa6b7c8f4ec071831b7"} Jul 14 19:47:19.254495 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.254035 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:19.255382 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.255364 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-99jl6" event={"ID":"f83805c3-112a-441e-a06b-dd01f9a1f2b1","Type":"ContainerStarted","Data":"91e4ef5ab802a0a8da0cf30d9b28411dcbabd2e19d8fc6c35472c41fd3b207ac"} Jul 14 19:47:19.256653 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.256633 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9zs2" event={"ID":"d35a5a8b-76ca-4120-9695-8e090145d939","Type":"ContainerStarted","Data":"ac57a5b28563e86d90be8fa13ced256b5cf8a033bf7b0eebb4858f065423de90"} Jul 14 19:47:19.272015 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.271975 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pcsfp" podStartSLOduration=1.266790066 podStartE2EDuration="7.27196346s" totalImagesPullingTime="6.005173394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:19.270878657 +0000 UTC m=+44.719505101" watchObservedRunningTime="2026-07-14 19:47:19.27196346 +0000 UTC m=+44.720589883" Jul 14 19:47:19.287136 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:19.287055 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-99jl6" podStartSLOduration=2.246659224 podStartE2EDuration="8.287041974s" totalImagesPullingTime="6.04038275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:19.286823023 +0000 UTC m=+44.735449467" watchObservedRunningTime="2026-07-14 19:47:19.287041974 +0000 UTC m=+44.735668421" Jul 14 19:47:21.264540 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:21.264462 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9zs2" event={"ID":"d35a5a8b-76ca-4120-9695-8e090145d939","Type":"ContainerStarted","Data":"b51f58a5217dd103edb06649ef38888b2ab2e3c60c0894b9341bdefc3a9586aa"} Jul 14 19:47:21.662813 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:21.662772 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:21.727887 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:21.727853 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:22.064000 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:22.063910 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:22.153113 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:22.153078 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:24.275795 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:24.275754 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9zs2" event={"ID":"d35a5a8b-76ca-4120-9695-8e090145d939","Type":"ContainerStarted","Data":"5fa65d3b7f81bd1d295db7f6ec78cd44ad1320c23406579c3ba6e6cd32648dd1"} Jul 14 19:47:24.298188 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:24.298134 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k9zs2" podStartSLOduration=2.8248911039999998 podStartE2EDuration="13.298118787s" totalImagesPullingTime="10.473227683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:24.297919631 +0000 UTC m=+49.746546076" watchObservedRunningTime="2026-07-14 19:47:24.298118787 +0000 UTC m=+49.746745231" Jul 14 19:47:27.685173 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:27.685144 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:27.786455 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:27.786425 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:28.077590 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.077507 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:28.180338 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.180307 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:28.291765 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.291737 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:28.299200 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.299178 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf"] Jul 14 19:47:28.303094 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.303077 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:28.305822 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.305802 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dx26f\"" Jul 14 19:47:28.305922 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.305801 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 14 19:47:28.311071 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.311036 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf"] Jul 14 19:47:28.393856 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.393823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25e85cb9-250c-429b-bc86-3a8a0e7bfac6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7zhwf\" (UID: \"25e85cb9-250c-429b-bc86-3a8a0e7bfac6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:28.495149 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.495110 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25e85cb9-250c-429b-bc86-3a8a0e7bfac6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7zhwf\" (UID: \"25e85cb9-250c-429b-bc86-3a8a0e7bfac6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:28.495316 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:28.495243 2553 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 14 19:47:28.495316 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:28.495306 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e85cb9-250c-429b-bc86-3a8a0e7bfac6-tls-certificates podName:25e85cb9-250c-429b-bc86-3a8a0e7bfac6 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:28.995292429 +0000 UTC m=+54.443918852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/25e85cb9-250c-429b-bc86-3a8a0e7bfac6-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-7zhwf" (UID: "25e85cb9-250c-429b-bc86-3a8a0e7bfac6") : secret "prometheus-operator-admission-webhook-tls" not found Jul 14 19:47:28.847114 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.847015 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:28.998826 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:28.998778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25e85cb9-250c-429b-bc86-3a8a0e7bfac6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7zhwf\" (UID: \"25e85cb9-250c-429b-bc86-3a8a0e7bfac6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:29.002075 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:29.002027 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25e85cb9-250c-429b-bc86-3a8a0e7bfac6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7zhwf\" (UID: \"25e85cb9-250c-429b-bc86-3a8a0e7bfac6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:29.212678 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:29.212642 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:29.262230 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:29.262202 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcsfp" Jul 14 19:47:29.334967 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:29.334929 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf"] Jul 14 19:47:29.339250 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:29.339223 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e85cb9_250c_429b_bc86_3a8a0e7bfac6.slice/crio-218fd48cb8f19cfae012e8df74ddbc7d8771c84f2e3a59dfc92d5a9fedbc1f4d WatchSource:0}: Error finding container 218fd48cb8f19cfae012e8df74ddbc7d8771c84f2e3a59dfc92d5a9fedbc1f4d: Status 404 returned error can't find the container with id 218fd48cb8f19cfae012e8df74ddbc7d8771c84f2e3a59dfc92d5a9fedbc1f4d Jul 14 19:47:30.294247 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:30.294214 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" event={"ID":"25e85cb9-250c-429b-bc86-3a8a0e7bfac6","Type":"ContainerStarted","Data":"218fd48cb8f19cfae012e8df74ddbc7d8771c84f2e3a59dfc92d5a9fedbc1f4d"} Jul 14 19:47:32.341201 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:32.341167 2553 patch_prober.go:28] interesting pod/image-registry-8f854fddd-dtm2f container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:47:32.341571 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:32.341219 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" podUID="a8249e15-4445-4b01-a11e-fe7a705b62b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:47:33.813970 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:33.813936 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:33.894469 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:33.894438 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:35.186220 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.186193 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddrpt" Jul 14 19:47:35.246041 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.246016 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8f854fddd-dtm2f" Jul 14 19:47:35.311916 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.311882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" event={"ID":"25e85cb9-250c-429b-bc86-3a8a0e7bfac6","Type":"ContainerStarted","Data":"6893b327dc7c1fd28af0d43c96e5b7a83d647bba25b0604b936b7fcbae89b4f9"} Jul 14 19:47:35.312120 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.312054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:35.316735 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.316711 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" Jul 14 19:47:35.322333 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.322306 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:35.350104 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.350040 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7zhwf" podStartSLOduration=1.965443979 podStartE2EDuration="7.350026172s" totalImagesPullingTime="5.384582193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:35.332085815 +0000 UTC m=+60.780712261" watchObservedRunningTime="2026-07-14 19:47:35.350026172 +0000 UTC m=+60.798652617" Jul 14 19:47:35.421265 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.421238 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:35.487376 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:35.487301 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:36.181375 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:36.181347 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:36.309812 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:36.309786 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:36.390103 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:36.390078 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:36.595688 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:36.595607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:36.665011 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:36.664982 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:36.798713 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:36.798681 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:37.198732 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:37.198698 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:38.154502 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:38.154471 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:38.225317 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:38.225289 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:38.417548 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:38.417452 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:38.518894 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:38.518854 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:39.774033 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.774002 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:39.774476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.774046 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:39.776535 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.776510 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:47:39.777254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.777241 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:47:39.787277 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.787246 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/437754cb-a459-4410-b75f-8ee2f10721ba-original-pull-secret\") pod \"global-pull-secret-syncer-26pgr\" (UID: \"437754cb-a459-4410-b75f-8ee2f10721ba\") " pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:39.787424 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.787346 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebae7f13-3078-42b4-b221-aa64593e0bfb-metrics-certs\") pod \"network-metrics-daemon-hj4v9\" (UID: \"ebae7f13-3078-42b4-b221-aa64593e0bfb\") " pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:39.875081 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.875029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:39.877281 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.877263 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:47:39.888316 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.888292 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:47:39.898541 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.898510 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trllt\" (UniqueName: \"kubernetes.io/projected/197f9801-26a1-4614-9ef8-9e89a67bdb1d-kube-api-access-trllt\") pod \"network-check-target-8fbqx\" (UID: \"197f9801-26a1-4614-9ef8-9e89a67bdb1d\") " pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:39.958835 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.958797 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26pgr" Jul 14 19:47:39.966531 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.966509 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvlkh\"" Jul 14 19:47:39.970538 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.970520 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xm9xl\"" Jul 14 19:47:39.974532 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.974514 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hj4v9" Jul 14 19:47:39.979364 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:39.979342 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:40.129282 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.129251 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-26pgr"] Jul 14 19:47:40.132193 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:40.132170 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437754cb_a459_4410_b75f_8ee2f10721ba.slice/crio-1f51e24915a014694042d36fefa823f0862c045cecf253d306ac9b85faa6ce67 WatchSource:0}: Error finding container 1f51e24915a014694042d36fefa823f0862c045cecf253d306ac9b85faa6ce67: Status 404 returned error can't find the container with id 1f51e24915a014694042d36fefa823f0862c045cecf253d306ac9b85faa6ce67 Jul 14 19:47:40.142665 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.142628 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8fbqx"] Jul 14 19:47:40.145216 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.145196 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hj4v9"] Jul 14 19:47:40.147384 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:40.147361 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197f9801_26a1_4614_9ef8_9e89a67bdb1d.slice/crio-6597b027949fbce25df1d3dd82d1d54f5291f5c97bcdbdf739afcb7bc5510a09 WatchSource:0}: Error finding container 6597b027949fbce25df1d3dd82d1d54f5291f5c97bcdbdf739afcb7bc5510a09: Status 404 returned error can't find the container with id 6597b027949fbce25df1d3dd82d1d54f5291f5c97bcdbdf739afcb7bc5510a09 Jul 14 19:47:40.148218 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:40.148199 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebae7f13_3078_42b4_b221_aa64593e0bfb.slice/crio-9b02e0400e930ea87efaf83eb6b789402e04905bd785c48dc561f048326750d4 WatchSource:0}: Error finding container 9b02e0400e930ea87efaf83eb6b789402e04905bd785c48dc561f048326750d4: Status 404 returned error can't find the container with id 9b02e0400e930ea87efaf83eb6b789402e04905bd785c48dc561f048326750d4 Jul 14 19:47:40.321983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.321909 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:40.327705 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.327680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hj4v9" event={"ID":"ebae7f13-3078-42b4-b221-aa64593e0bfb","Type":"ContainerStarted","Data":"9b02e0400e930ea87efaf83eb6b789402e04905bd785c48dc561f048326750d4"} Jul 14 19:47:40.328731 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.328712 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8fbqx" event={"ID":"197f9801-26a1-4614-9ef8-9e89a67bdb1d","Type":"ContainerStarted","Data":"6597b027949fbce25df1d3dd82d1d54f5291f5c97bcdbdf739afcb7bc5510a09"} Jul 14 19:47:40.329755 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:40.329737 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-26pgr" event={"ID":"437754cb-a459-4410-b75f-8ee2f10721ba","Type":"ContainerStarted","Data":"1f51e24915a014694042d36fefa823f0862c045cecf253d306ac9b85faa6ce67"} Jul 14 19:47:42.338805 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.338717 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hj4v9" event={"ID":"ebae7f13-3078-42b4-b221-aa64593e0bfb","Type":"ContainerStarted","Data":"468b40f3ca6798d81eb63fa6da6ae5e24b7f64cb20ca99709953e40327788e4f"} Jul 14 19:47:42.338805 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.338774 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hj4v9" event={"ID":"ebae7f13-3078-42b4-b221-aa64593e0bfb","Type":"ContainerStarted","Data":"5fe699c17ba1aa23311346b89fe463e64545df2e077574cc1e9e389ab401e717"} Jul 14 19:47:42.357399 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.356391 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hj4v9" podStartSLOduration=65.440645554 podStartE2EDuration="1m7.356373525s" totalImagesPullingTime="1.915727971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:42.355989399 +0000 UTC m=+67.804615845" watchObservedRunningTime="2026-07-14 19:47:42.356373525 +0000 UTC m=+67.804999971" Jul 14 19:47:42.498756 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.498702 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:42.602481 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.602406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:42.765035 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.764998 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:42.785683 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.785648 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j"] Jul 14 19:47:42.788636 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.788611 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.791042 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.790987 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 19:47:42.791166 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.791118 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 14 19:47:42.791234 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.791007 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 14 19:47:42.791403 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.791362 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 19:47:42.792362 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.792085 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9bwhh\"" Jul 14 19:47:42.792362 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.792109 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 19:47:42.800432 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.800409 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j"] Jul 14 19:47:42.842766 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.842731 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p5lkw"] Jul 14 19:47:42.845085 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.845024 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.847675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.847653 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6x7x8\"" Jul 14 19:47:42.848029 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.848001 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 19:47:42.848151 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.848028 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 19:47:42.848215 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.848205 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 19:47:42.869940 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.869868 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:42.896104 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896050 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-sys\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896153 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmlz\" (UniqueName: \"kubernetes.io/projected/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-kube-api-access-4hmlz\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896197 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.896297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896242 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-tls\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896277 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1206c5d-c330-49e1-89ed-e5a98880fb3a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.896489 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896489 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896358 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-metrics-client-ca\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896489 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896410 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-root\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896489 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896443 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.896688 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896505 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896751 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896662 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwfl\" (UniqueName: \"kubernetes.io/projected/b1206c5d-c330-49e1-89ed-e5a98880fb3a-kube-api-access-9cwfl\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.896801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896772 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-textfile\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.896851 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.896806 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-wtmp\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.947205 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.947166 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:42.998122 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998082 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwfl\" (UniqueName: \"kubernetes.io/projected/b1206c5d-c330-49e1-89ed-e5a98880fb3a-kube-api-access-9cwfl\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-textfile\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998170 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-wtmp\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998201 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-sys\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998228 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmlz\" (UniqueName: \"kubernetes.io/projected/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-kube-api-access-4hmlz\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998255 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998289 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-tls\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998323 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1206c5d-c330-49e1-89ed-e5a98880fb3a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998346 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998390 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998369 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-metrics-client-ca\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998802 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998404 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-root\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.998802 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998432 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:42.998802 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.998458 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999104 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:42.999074 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 19:47:42.999189 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999119 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999189 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:42.999153 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-tls podName:837d41a4-fb76-4673-a1e6-67c5a9a3bca2 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:43.49913387 +0000 UTC m=+68.947760300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-tls") pod "node-exporter-p5lkw" (UID: "837d41a4-fb76-4673-a1e6-67c5a9a3bca2") : secret "node-exporter-tls" not found Jul 14 19:47:42.999294 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999193 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-root\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999294 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999250 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-sys\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999387 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-wtmp\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999387 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:42.999361 2553 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 14 19:47:42.999490 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:47:42.999441 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-tls podName:b1206c5d-c330-49e1-89ed-e5a98880fb3a nodeName:}" failed. No retries permitted until 2026-07-14 19:47:43.499423243 +0000 UTC m=+68.948049670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-5lc9j" (UID: "b1206c5d-c330-49e1-89ed-e5a98880fb3a") : secret "openshift-state-metrics-tls" not found Jul 14 19:47:42.999762 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-metrics-client-ca\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999762 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999742 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-textfile\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:42.999925 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:42.999911 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1206c5d-c330-49e1-89ed-e5a98880fb3a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:43.002057 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.002031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:43.002161 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.002083 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:43.014834 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.014795 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwfl\" (UniqueName: \"kubernetes.io/projected/b1206c5d-c330-49e1-89ed-e5a98880fb3a-kube-api-access-9cwfl\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:43.014990 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.014962 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmlz\" (UniqueName: \"kubernetes.io/projected/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-kube-api-access-4hmlz\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:43.075219 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.075153 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.172562 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.172480 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.271964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.271922 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.372225 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.372130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.472638 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.472570 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.502680 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.502640 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:43.502894 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.502696 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-tls\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:43.505116 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.505092 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/837d41a4-fb76-4673-a1e6-67c5a9a3bca2-node-exporter-tls\") pod \"node-exporter-p5lkw\" (UID: \"837d41a4-fb76-4673-a1e6-67c5a9a3bca2\") " pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:43.505255 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.505168 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1206c5d-c330-49e1-89ed-e5a98880fb3a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lc9j\" (UID: \"b1206c5d-c330-49e1-89ed-e5a98880fb3a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:43.607331 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.607279 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.701141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.701111 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" Jul 14 19:47:43.754945 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.754911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5lkw" Jul 14 19:47:43.765817 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.765748 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.839204 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.839170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j"] Jul 14 19:47:43.843586 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:43.843557 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1206c5d_c330_49e1_89ed_e5a98880fb3a.slice/crio-e70ab28d7c3863694099180ec6d0497cb2a5cbdb0d7c91cc8ef827e6bebf6080 WatchSource:0}: Error finding container e70ab28d7c3863694099180ec6d0497cb2a5cbdb0d7c91cc8ef827e6bebf6080: Status 404 returned error can't find the container with id e70ab28d7c3863694099180ec6d0497cb2a5cbdb0d7c91cc8ef827e6bebf6080 Jul 14 19:47:43.850200 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.850049 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:47:43.850472 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.850397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:43.853976 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.853959 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.856377 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856352 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 19:47:43.856801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856761 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 19:47:43.856801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856774 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 19:47:43.856801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856813 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 19:47:43.857013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856867 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wn7wm\"" Jul 14 19:47:43.857013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856765 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 19:47:43.857013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856950 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 19:47:43.857013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856979 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 19:47:43.857013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.856987 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 19:47:43.862346 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.862222 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 19:47:43.880312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.880282 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:47:43.905425 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905397 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwk7g\" (UniqueName: \"kubernetes.io/projected/dbbc5f1a-4050-4feb-abdf-264e33e700d3-kube-api-access-kwk7g\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905466 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905496 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905517 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905538 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dbbc5f1a-4050-4feb-abdf-264e33e700d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905559 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbbc5f1a-4050-4feb-abdf-264e33e700d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905579 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbbc5f1a-4050-4feb-abdf-264e33e700d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905593 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905632 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905674 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbbc5f1a-4050-4feb-abdf-264e33e700d3-config-out\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905702 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905744 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbbc5f1a-4050-4feb-abdf-264e33e700d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:43.905772 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:43.905771 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-web-config\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006487 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006645 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006690 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006640 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006736 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006688 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dbbc5f1a-4050-4feb-abdf-264e33e700d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006736 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbbc5f1a-4050-4feb-abdf-264e33e700d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006840 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006747 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbbc5f1a-4050-4feb-abdf-264e33e700d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006840 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006772 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006840 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.006840 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006828 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbbc5f1a-4050-4feb-abdf-264e33e700d3-config-out\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.007024 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006852 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.007024 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006899 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbbc5f1a-4050-4feb-abdf-264e33e700d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.007024 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006927 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-web-config\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.007024 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.006970 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwk7g\" (UniqueName: \"kubernetes.io/projected/dbbc5f1a-4050-4feb-abdf-264e33e700d3-kube-api-access-kwk7g\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.007257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.007098 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dbbc5f1a-4050-4feb-abdf-264e33e700d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.007841 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.007784 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbbc5f1a-4050-4feb-abdf-264e33e700d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.009716 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.009689 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbbc5f1a-4050-4feb-abdf-264e33e700d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.010641 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.010591 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.011386 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.011009 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.011386 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.011104 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.011386 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.011149 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbbc5f1a-4050-4feb-abdf-264e33e700d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.011670 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.011435 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbbc5f1a-4050-4feb-abdf-264e33e700d3-config-out\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.011950 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.011907 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.012574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.012339 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-web-config\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.012574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.012538 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.013149 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.013032 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dbbc5f1a-4050-4feb-abdf-264e33e700d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.015835 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.015813 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwk7g\" (UniqueName: \"kubernetes.io/projected/dbbc5f1a-4050-4feb-abdf-264e33e700d3-kube-api-access-kwk7g\") pod \"alertmanager-main-0\" (UID: \"dbbc5f1a-4050-4feb-abdf-264e33e700d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.063692 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.063652 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:44.166953 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.166910 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:47:44.333083 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.333033 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:47:44.337089 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:44.337042 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbc5f1a_4050_4feb_abdf_264e33e700d3.slice/crio-d2a0516e23cfdcc41e9c2a212fb6a4c9d246bca74b488e36ad0efef8e0087c5c WatchSource:0}: Error finding container d2a0516e23cfdcc41e9c2a212fb6a4c9d246bca74b488e36ad0efef8e0087c5c: Status 404 returned error can't find the container with id d2a0516e23cfdcc41e9c2a212fb6a4c9d246bca74b488e36ad0efef8e0087c5c Jul 14 19:47:44.349118 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.349078 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" event={"ID":"b1206c5d-c330-49e1-89ed-e5a98880fb3a","Type":"ContainerStarted","Data":"ce0f8c7579a177c0199a696a853562fd87f9bd46189f0da40dd77408eb1ed061"} Jul 14 19:47:44.349283 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.349151 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" event={"ID":"b1206c5d-c330-49e1-89ed-e5a98880fb3a","Type":"ContainerStarted","Data":"00d6d6ddd22aad93220181b44873edbecea3c59ea40583533d1e1480a7504f90"} Jul 14 19:47:44.349283 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.349178 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" event={"ID":"b1206c5d-c330-49e1-89ed-e5a98880fb3a","Type":"ContainerStarted","Data":"e70ab28d7c3863694099180ec6d0497cb2a5cbdb0d7c91cc8ef827e6bebf6080"} Jul 14 19:47:44.350427 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.350395 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5lkw" event={"ID":"837d41a4-fb76-4673-a1e6-67c5a9a3bca2","Type":"ContainerStarted","Data":"03d96c4562d233459c51278785baae532926ffafd7f78c29646cc3c01b02b15f"} Jul 14 19:47:44.352406 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.352378 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8fbqx" event={"ID":"197f9801-26a1-4614-9ef8-9e89a67bdb1d","Type":"ContainerStarted","Data":"f5d3c6cc2e3edb39820643cad632e5f29da43406b449298c16933266e0929137"} Jul 14 19:47:44.352704 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.352674 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:47:44.354344 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.354322 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"d2a0516e23cfdcc41e9c2a212fb6a4c9d246bca74b488e36ad0efef8e0087c5c"} Jul 14 19:47:44.373297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:44.372832 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8fbqx" podStartSLOduration=65.803250095 podStartE2EDuration="1m9.372809525s" totalImagesPullingTime="3.56955943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:44.371545744 +0000 UTC m=+69.820172191" watchObservedRunningTime="2026-07-14 19:47:44.372809525 +0000 UTC m=+69.821435971" Jul 14 19:47:45.275315 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:45.275278 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:45.569319 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:45.569243 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:45.742144 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:45.742095 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:45.845858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:45.845770 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:45.922237 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:45.922204 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:47.367247 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.367210 2553 generic.go:356] "Generic (PLEG): container finished" podID="837d41a4-fb76-4673-a1e6-67c5a9a3bca2" containerID="c656ce31d29d434428edf54d951a78389b79f56d623d4b5709277e6b8ca22357" exitCode=0 Jul 14 19:47:47.367638 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.367293 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5lkw" event={"ID":"837d41a4-fb76-4673-a1e6-67c5a9a3bca2","Type":"ContainerDied","Data":"c656ce31d29d434428edf54d951a78389b79f56d623d4b5709277e6b8ca22357"} Jul 14 19:47:47.368740 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.368713 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-26pgr" event={"ID":"437754cb-a459-4410-b75f-8ee2f10721ba","Type":"ContainerStarted","Data":"987fef4151c27ab49687d939b27e2c6634c5dc194069ddf1b10e83d9dafd2d2e"} Jul 14 19:47:47.370468 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.370429 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5688c564b9-qdd86"] Jul 14 19:47:47.387664 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.387639 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5688c564b9-qdd86"] Jul 14 19:47:47.387813 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.387730 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.390297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.390270 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 14 19:47:47.390470 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.390280 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 14 19:47:47.390470 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.390333 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5u5u2rb5qvbhh\"" Jul 14 19:47:47.390470 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.390376 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 19:47:47.390810 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.390547 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-chpp7\"" Jul 14 19:47:47.390810 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.390646 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 14 19:47:47.433272 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.433122 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-26pgr" podStartSLOduration=66.169989708 podStartE2EDuration="1m12.433105684s" totalImagesPullingTime="6.263115976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:47.402365654 +0000 UTC m=+72.850992100" watchObservedRunningTime="2026-07-14 19:47:47.433105684 +0000 UTC m=+72.881732129" Jul 14 19:47:47.436126 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436105 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nld\" (UniqueName: \"kubernetes.io/projected/541fea56-0550-43d7-9a8e-5fbf794af13c-kube-api-access-b9nld\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.436223 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436158 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-secret-metrics-server-client-certs\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.436223 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436186 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-secret-metrics-server-tls\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.436322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436243 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/541fea56-0550-43d7-9a8e-5fbf794af13c-audit-log\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.436322 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436266 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/541fea56-0550-43d7-9a8e-5fbf794af13c-metrics-server-audit-profiles\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.436417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436328 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/541fea56-0550-43d7-9a8e-5fbf794af13c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.436466 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.436441 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-client-ca-bundle\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537248 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537208 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/541fea56-0550-43d7-9a8e-5fbf794af13c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537248 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537251 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-client-ca-bundle\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537305 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nld\" (UniqueName: \"kubernetes.io/projected/541fea56-0550-43d7-9a8e-5fbf794af13c-kube-api-access-b9nld\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537324 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-secret-metrics-server-client-certs\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537342 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-secret-metrics-server-tls\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537363 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/541fea56-0550-43d7-9a8e-5fbf794af13c-audit-log\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537377 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/541fea56-0550-43d7-9a8e-5fbf794af13c-metrics-server-audit-profiles\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537846 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537791 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/541fea56-0550-43d7-9a8e-5fbf794af13c-audit-log\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.537998 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.537973 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/541fea56-0550-43d7-9a8e-5fbf794af13c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.538526 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.538507 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/541fea56-0550-43d7-9a8e-5fbf794af13c-metrics-server-audit-profiles\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.539867 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.539844 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-secret-metrics-server-tls\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.540179 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.540164 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-client-ca-bundle\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.540239 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.540223 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/541fea56-0550-43d7-9a8e-5fbf794af13c-secret-metrics-server-client-certs\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.546357 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.546330 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:47.554414 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.554368 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nld\" (UniqueName: \"kubernetes.io/projected/541fea56-0550-43d7-9a8e-5fbf794af13c-kube-api-access-b9nld\") pod \"metrics-server-5688c564b9-qdd86\" (UID: \"541fea56-0550-43d7-9a8e-5fbf794af13c\") " pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.555278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.555258 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5974b4cf99-bd6fc"] Jul 14 19:47:47.573042 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.573010 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd"] Jul 14 19:47:47.573206 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.573172 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.576082 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576039 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:47:47.576082 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576023 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 19:47:47.576289 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576115 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 19:47:47.576289 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576035 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:47:47.576289 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576049 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h5xm7\"" Jul 14 19:47:47.576482 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576466 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 19:47:47.576581 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576565 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 19:47:47.576860 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.576839 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 19:47:47.581308 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.581290 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 19:47:47.593857 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.593833 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5974b4cf99-bd6fc"] Jul 14 19:47:47.593969 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.593871 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd"] Jul 14 19:47:47.593969 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.593945 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.596623 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.596569 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2crds\"" Jul 14 19:47:47.596623 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.596569 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 19:47:47.614094 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.614056 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:47.638233 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638200 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-trusted-ca-bundle\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638326 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638249 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-config\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638326 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-oauth-serving-cert\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638326 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638299 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-service-ca\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638463 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638339 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4t8\" (UniqueName: \"kubernetes.io/projected/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-kube-api-access-qj4t8\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638463 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638372 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-oauth-config\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638463 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638390 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3e063e-0d08-4f83-a521-2aa4bfa27004-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-ldjhd\" (UID: \"0e3e063e-0d08-4f83-a521-2aa4bfa27004\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.638463 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638432 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-serving-cert\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.638463 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.638448 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjhz\" (UniqueName: \"kubernetes.io/projected/0e3e063e-0d08-4f83-a521-2aa4bfa27004-kube-api-access-2wjhz\") pod \"monitoring-plugin-5f68df7f58-ldjhd\" (UID: \"0e3e063e-0d08-4f83-a521-2aa4bfa27004\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.717186 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.717160 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:47:47.718171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.718151 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:47.738873 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.738844 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-trusted-ca-bundle\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739009 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.738904 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-config\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739009 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.738932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-oauth-serving-cert\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739009 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.738961 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-service-ca\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739009 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.739002 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4t8\" (UniqueName: \"kubernetes.io/projected/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-kube-api-access-qj4t8\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.739027 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-oauth-config\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.739056 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3e063e-0d08-4f83-a521-2aa4bfa27004-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-ldjhd\" (UID: \"0e3e063e-0d08-4f83-a521-2aa4bfa27004\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.739240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.739139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-serving-cert\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.739240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.739162 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjhz\" (UniqueName: \"kubernetes.io/projected/0e3e063e-0d08-4f83-a521-2aa4bfa27004-kube-api-access-2wjhz\") pod \"monitoring-plugin-5f68df7f58-ldjhd\" (UID: \"0e3e063e-0d08-4f83-a521-2aa4bfa27004\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.740073 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.739673 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-config\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.740182 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.740134 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-oauth-serving-cert\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.740260 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.740210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-trusted-ca-bundle\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.740607 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.740565 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-service-ca\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.742089 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.742053 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-oauth-config\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.742382 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.742363 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e3e063e-0d08-4f83-a521-2aa4bfa27004-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-ldjhd\" (UID: \"0e3e063e-0d08-4f83-a521-2aa4bfa27004\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.742525 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.742507 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-serving-cert\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.748895 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.748874 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjhz\" (UniqueName: \"kubernetes.io/projected/0e3e063e-0d08-4f83-a521-2aa4bfa27004-kube-api-access-2wjhz\") pod \"monitoring-plugin-5f68df7f58-ldjhd\" (UID: \"0e3e063e-0d08-4f83-a521-2aa4bfa27004\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.749830 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.749809 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4t8\" (UniqueName: \"kubernetes.io/projected/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-kube-api-access-qj4t8\") pod \"console-5974b4cf99-bd6fc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.825212 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.822542 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:47.883154 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.883126 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:47:47.885283 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.885246 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5688c564b9-qdd86"] Jul 14 19:47:47.901801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.901781 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:47.903864 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:47.903839 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541fea56_0550_43d7_9a8e_5fbf794af13c.slice/crio-e610598eb1f3a29aac54427435ca7db9983549116ba6816caa3cd56130d29c10 WatchSource:0}: Error finding container e610598eb1f3a29aac54427435ca7db9983549116ba6816caa3cd56130d29c10: Status 404 returned error can't find the container with id e610598eb1f3a29aac54427435ca7db9983549116ba6816caa3cd56130d29c10 Jul 14 19:47:47.954948 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:47.954858 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:48.003412 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.003376 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp"] Jul 14 19:47:48.033057 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.032989 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp"] Jul 14 19:47:48.033240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.033153 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.036666 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.036631 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5974b4cf99-bd6fc"] Jul 14 19:47:48.036960 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.036943 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 14 19:47:48.037143 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.037125 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9fjcs\"" Jul 14 19:47:48.037427 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.037330 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 14 19:47:48.037427 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.037378 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 14 19:47:48.037538 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.037333 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 14 19:47:48.037538 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.037481 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 14 19:47:48.040687 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:48.040666 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2dfd38_8737_4c7d_ac6b_136b479ee3bc.slice/crio-69f6c4dd38f38d7c0d07adacd6870ae8679335aa09e83f1ca1a7d73101b22155 WatchSource:0}: Error finding container 69f6c4dd38f38d7c0d07adacd6870ae8679335aa09e83f1ca1a7d73101b22155: Status 404 returned error can't find the container with id 69f6c4dd38f38d7c0d07adacd6870ae8679335aa09e83f1ca1a7d73101b22155 Jul 14 19:47:48.041862 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.041775 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 14 19:47:48.075948 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.075918 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd"] Jul 14 19:47:48.079079 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:48.079031 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3e063e_0d08_4f83_a521_2aa4bfa27004.slice/crio-1ee949425a698df31da2057644a0c74efd339e4bfff9077a0f309e563ee5c1d8 WatchSource:0}: Error finding container 1ee949425a698df31da2057644a0c74efd339e4bfff9077a0f309e563ee5c1d8: Status 404 returned error can't find the container with id 1ee949425a698df31da2057644a0c74efd339e4bfff9077a0f309e563ee5c1d8 Jul 14 19:47:48.088511 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.088487 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:48.141726 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141682 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-serving-certs-ca-bundle\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.141908 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-metrics-client-ca\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.141908 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141757 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lh6j\" (UniqueName: \"kubernetes.io/projected/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-kube-api-access-6lh6j\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.141908 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141804 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-telemeter-client-tls\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.141908 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141869 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-secret-telemeter-client\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.141908 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141902 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-federate-client-tls\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.142135 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141941 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.142135 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.141970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.180048 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.180017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:48.242918 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.242881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-federate-client-tls\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.242925 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.242947 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.242983 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-serving-certs-ca-bundle\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.243004 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-metrics-client-ca\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.243019 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lh6j\" (UniqueName: \"kubernetes.io/projected/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-kube-api-access-6lh6j\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.243048 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-telemeter-client-tls\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243363 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.243129 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-secret-telemeter-client\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.243906 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.243863 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-serving-certs-ca-bundle\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.244040 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.243925 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-metrics-client-ca\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.244128 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.244057 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.245447 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.245419 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-secret-telemeter-client\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.245547 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.245452 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.245738 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.245720 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-federate-client-tls\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.245770 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.245747 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-telemeter-client-tls\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.253841 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.253821 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lh6j\" (UniqueName: \"kubernetes.io/projected/6f71e1e0-4c20-42a2-8fd0-a5f71ea399de-kube-api-access-6lh6j\") pod \"telemeter-client-d4f9f74d7-2qdfp\" (UID: \"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de\") " pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.347825 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.347784 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" Jul 14 19:47:48.373023 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.372989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974b4cf99-bd6fc" event={"ID":"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc","Type":"ContainerStarted","Data":"69f6c4dd38f38d7c0d07adacd6870ae8679335aa09e83f1ca1a7d73101b22155"} Jul 14 19:47:48.375036 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.375007 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5lkw" event={"ID":"837d41a4-fb76-4673-a1e6-67c5a9a3bca2","Type":"ContainerStarted","Data":"a8666c035ed24e9a97632b0df98fad7cb01abd066d1d669f99c5e62a3b0d278d"} Jul 14 19:47:48.375179 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.375052 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5lkw" event={"ID":"837d41a4-fb76-4673-a1e6-67c5a9a3bca2","Type":"ContainerStarted","Data":"61052b61c2e617a4cabf650b26621decc5189a31dbdb3bacfc4e99c169b1a74d"} Jul 14 19:47:48.376491 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.376470 2553 generic.go:356] "Generic (PLEG): container finished" podID="dbbc5f1a-4050-4feb-abdf-264e33e700d3" containerID="258f36ff7fa7fc2ce18412de8b424f9ce40d242fef29fcf917a42ac7408613a5" exitCode=0 Jul 14 19:47:48.376597 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.376539 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerDied","Data":"258f36ff7fa7fc2ce18412de8b424f9ce40d242fef29fcf917a42ac7408613a5"} Jul 14 19:47:48.378954 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.378933 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" event={"ID":"b1206c5d-c330-49e1-89ed-e5a98880fb3a","Type":"ContainerStarted","Data":"b7ddccbe1e79b7e803a11f0c046a1f39f0cc7a956aec630b90a74ce3e87fe0eb"} Jul 14 19:47:48.380264 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.380238 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" event={"ID":"0e3e063e-0d08-4f83-a521-2aa4bfa27004","Type":"ContainerStarted","Data":"1ee949425a698df31da2057644a0c74efd339e4bfff9077a0f309e563ee5c1d8"} Jul 14 19:47:48.381449 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.381385 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" event={"ID":"541fea56-0550-43d7-9a8e-5fbf794af13c","Type":"ContainerStarted","Data":"e610598eb1f3a29aac54427435ca7db9983549116ba6816caa3cd56130d29c10"} Jul 14 19:47:48.394347 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.394249 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p5lkw" podStartSLOduration=3.682394924 podStartE2EDuration="6.394227799s" totalImagesPullingTime="2.614535581s" totalInitContainerRuntime="97.297294ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:48.393170863 +0000 UTC m=+73.841797330" watchObservedRunningTime="2026-07-14 19:47:48.394227799 +0000 UTC m=+73.842854243" Jul 14 19:47:48.411833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.411780 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lc9j" podStartSLOduration=3.046001308 podStartE2EDuration="6.411762631s" totalImagesPullingTime="3.365761323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:48.410607152 +0000 UTC m=+73.859233619" watchObservedRunningTime="2026-07-14 19:47:48.411762631 +0000 UTC m=+73.860389075" Jul 14 19:47:48.510699 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.510667 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp"] Jul 14 19:47:48.513876 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:47:48.513848 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f71e1e0_4c20_42a2_8fd0_a5f71ea399de.slice/crio-1bed6e306e3ccd8026ac8bc0a4c46caf01cd776cb7907e5dfe34b8311d491f54 WatchSource:0}: Error finding container 1bed6e306e3ccd8026ac8bc0a4c46caf01cd776cb7907e5dfe34b8311d491f54: Status 404 returned error can't find the container with id 1bed6e306e3ccd8026ac8bc0a4c46caf01cd776cb7907e5dfe34b8311d491f54 Jul 14 19:47:48.620381 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.620336 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:48.720978 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:48.720880 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:49.224483 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:49.224449 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:49.386781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:49.386733 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" event={"ID":"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de","Type":"ContainerStarted","Data":"1bed6e306e3ccd8026ac8bc0a4c46caf01cd776cb7907e5dfe34b8311d491f54"} Jul 14 19:47:53.408723 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.408684 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974b4cf99-bd6fc" event={"ID":"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc","Type":"ContainerStarted","Data":"90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f"} Jul 14 19:47:53.411964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.411937 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"e7ca194e128e2d7c3598fe68c23f3bbeb278bbdbdd8a8253d935eb970ede54e7"} Jul 14 19:47:53.412108 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.412003 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"d04402c7f0f78f92d2f351a718904c752b683ef84f9b1fa56094aa8ab3278a17"} Jul 14 19:47:53.412108 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.412035 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"214001ccd91078328d483617506a872d8dde75c0173469205441d639bca2a6bc"} Jul 14 19:47:53.412108 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.412087 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"38bcaf94e99bb667b552fa0a4131438c29f26d70af4b287d0026ad8d205418a0"} Jul 14 19:47:53.412260 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.412122 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"d35146f5c0cc5d77c59fbf95b150c48f334ea9559f400e56cb9a2f20429bbf89"} Jul 14 19:47:53.414272 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.414246 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" event={"ID":"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de","Type":"ContainerStarted","Data":"4bad26625a2643169bc6a0dd295c9afb650f85fc8a4724d155d029a27cfbcfdd"} Jul 14 19:47:53.414515 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.414303 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" event={"ID":"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de","Type":"ContainerStarted","Data":"ab49b0d3b6a7f5841369792e2a682a458fb6cf6402ef4098d3d7dc7eabd41cf4"} Jul 14 19:47:53.414515 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.414331 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" event={"ID":"6f71e1e0-4c20-42a2-8fd0-a5f71ea399de","Type":"ContainerStarted","Data":"22bece592791594384466050360a9f71f0ad9e53dc49b3a51913f73bc0ee6ac0"} Jul 14 19:47:53.415741 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.415717 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" event={"ID":"0e3e063e-0d08-4f83-a521-2aa4bfa27004","Type":"ContainerStarted","Data":"c08111e2822757944c0064a11ad3642d3d38a1958fc33588d3abdfa2e82ecad7"} Jul 14 19:47:53.415911 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.415887 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:53.417424 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.417377 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" event={"ID":"541fea56-0550-43d7-9a8e-5fbf794af13c","Type":"ContainerStarted","Data":"47ab2ac20b0d7c1ace7403813290bf46f385aafef3c09ad6b257ea7b8fc4d73e"} Jul 14 19:47:53.420867 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.420852 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" Jul 14 19:47:53.428722 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.428684 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5974b4cf99-bd6fc" podStartSLOduration=1.617812272 podStartE2EDuration="6.428671851s" totalImagesPullingTime="4.810859579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:53.427186318 +0000 UTC m=+78.875812792" watchObservedRunningTime="2026-07-14 19:47:53.428671851 +0000 UTC m=+78.877298295" Jul 14 19:47:53.444769 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.444724 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" podStartSLOduration=1.953942794 podStartE2EDuration="6.444710746s" totalImagesPullingTime="4.490767952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:53.443414629 +0000 UTC m=+78.892041073" watchObservedRunningTime="2026-07-14 19:47:53.444710746 +0000 UTC m=+78.893337191" Jul 14 19:47:53.461128 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.461045 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ldjhd" podStartSLOduration=2.143988268 podStartE2EDuration="6.461026093s" totalImagesPullingTime="4.317037825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:53.46041219 +0000 UTC m=+78.909038671" watchObservedRunningTime="2026-07-14 19:47:53.461026093 +0000 UTC m=+78.909652538" Jul 14 19:47:53.486868 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:53.486809 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d4f9f74d7-2qdfp" podStartSLOduration=2.097340437 podStartE2EDuration="6.486755984s" totalImagesPullingTime="4.389415547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:53.484909487 +0000 UTC m=+78.933535964" watchObservedRunningTime="2026-07-14 19:47:53.486755984 +0000 UTC m=+78.935382430" Jul 14 19:47:54.373608 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:54.373570 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5974b4cf99-bd6fc"] Jul 14 19:47:54.563319 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:54.563280 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:55.158385 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:55.158344 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:55.221624 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:55.221586 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:55.432123 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:55.432020 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dbbc5f1a-4050-4feb-abdf-264e33e700d3","Type":"ContainerStarted","Data":"40ebce56fcec22f53ee6d4d9a60f6edb55a3039256bc86f78bc5735124155ebc"} Jul 14 19:47:55.481247 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:55.481188 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.936121194 podStartE2EDuration="12.481168287s" totalImagesPullingTime="9.523543592s" totalInitContainerRuntime="21.503501ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:55.481054718 +0000 UTC m=+80.929681221" watchObservedRunningTime="2026-07-14 19:47:55.481168287 +0000 UTC m=+80.929794740" Jul 14 19:47:56.364229 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:56.364196 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:56.424534 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:56.424501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:57.440525 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:57.440485 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:57.883447 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:47:57.883397 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:48:04.984464 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:04.984427 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:05.290459 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:05.290379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:07.717871 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:07.717831 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:48:07.718281 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:07.717897 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:48:15.361282 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:15.361249 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8fbqx" Jul 14 19:48:20.451076 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.451027 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5974b4cf99-bd6fc" podUID="bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" containerName="console" containerID="cri-o://90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f" gracePeriod=15 Jul 14 19:48:20.739535 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.739502 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:48:20.820425 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820393 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-oauth-serving-cert\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820588 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820436 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-serving-cert\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820588 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820460 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj4t8\" (UniqueName: \"kubernetes.io/projected/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-kube-api-access-qj4t8\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820588 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820487 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-service-ca\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820588 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820507 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-oauth-config\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820588 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820538 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-trusted-ca-bundle\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820588 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820558 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-config\") pod \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\" (UID: \"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc\") " Jul 14 19:48:20.820945 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.820881 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:20.821090 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.821018 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-config" (OuterVolumeSpecName: "console-config") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:20.821090 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.821006 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:20.821090 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.821035 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:20.822740 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.822709 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:20.822816 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.822748 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:20.822816 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.822752 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-kube-api-access-qj4t8" (OuterVolumeSpecName: "kube-api-access-qj4t8") pod "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" (UID: "bd2dfd38-8737-4c7d-ac6b-136b479ee3bc"). InnerVolumeSpecName "kube-api-access-qj4t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:20.921448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921408 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-oauth-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:20.921448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921440 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-trusted-ca-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:20.921448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921449 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:20.921448 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921457 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-oauth-serving-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:20.921706 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921467 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-console-serving-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:20.921706 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921475 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4t8\" (UniqueName: \"kubernetes.io/projected/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-kube-api-access-qj4t8\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:20.921706 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:20.921483 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:48:21.531834 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.531804 2553 generic.go:356] "Generic (PLEG): container finished" podID="bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" containerID="90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f" exitCode=0 Jul 14 19:48:21.532309 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.531848 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974b4cf99-bd6fc" Jul 14 19:48:21.532309 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.531886 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974b4cf99-bd6fc" event={"ID":"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc","Type":"ContainerDied","Data":"90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f"} Jul 14 19:48:21.532309 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.531935 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974b4cf99-bd6fc" event={"ID":"bd2dfd38-8737-4c7d-ac6b-136b479ee3bc","Type":"ContainerDied","Data":"69f6c4dd38f38d7c0d07adacd6870ae8679335aa09e83f1ca1a7d73101b22155"} Jul 14 19:48:21.532309 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.531956 2553 scope.go:117] "RemoveContainer" containerID="90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f" Jul 14 19:48:21.540259 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.540244 2553 scope.go:117] "RemoveContainer" containerID="90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f" Jul 14 19:48:21.540557 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:48:21.540536 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f\": container with ID starting with 90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f not found: ID does not exist" containerID="90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f" Jul 14 19:48:21.540620 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.540564 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f"} err="failed to get container status \"90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f\": rpc error: code = NotFound desc = could not find container \"90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f\": container with ID starting with 90082130e2f822fd5d609c1a8a5f903f6c376c893023e5e110b6e5f879cc6e2f not found: ID does not exist" Jul 14 19:48:21.550278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.550255 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5974b4cf99-bd6fc"] Jul 14 19:48:21.554400 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:21.554378 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5974b4cf99-bd6fc"] Jul 14 19:48:23.034895 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:23.034857 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" path="/var/lib/kubelet/pods/bd2dfd38-8737-4c7d-ac6b-136b479ee3bc/volumes" Jul 14 19:48:27.722991 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:27.722958 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:48:27.731402 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:27.731368 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5688c564b9-qdd86" Jul 14 19:48:27.957586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:27.957555 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:30.919527 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:30.919496 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:31.192812 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:31.192706 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:31.319359 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:31.319322 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:47.911699 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:47.911665 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:58.915941 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:58.915897 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:58.974367 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:58.974313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:59.914682 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.914648 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f95df6849-jmlpc"] Jul 14 19:48:59.914942 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.914931 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" containerName="console" Jul 14 19:48:59.915001 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.914944 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" containerName="console" Jul 14 19:48:59.915001 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.914998 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2dfd38-8737-4c7d-ac6b-136b479ee3bc" containerName="console" Jul 14 19:48:59.917717 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.917698 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:48:59.920417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.920386 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 19:48:59.921719 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.921698 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:48:59.921936 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.921906 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h5xm7\"" Jul 14 19:48:59.922252 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.922220 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:48:59.922343 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.922021 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 19:48:59.922852 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.922832 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 19:48:59.923708 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.923009 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 19:48:59.923708 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.923040 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 19:48:59.926957 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.926939 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 19:48:59.930180 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.930159 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f95df6849-jmlpc"] Jul 14 19:48:59.971542 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:48:59.971497 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:00.025968 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.025938 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-service-ca\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.026119 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.025976 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-serving-cert\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.026119 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.026001 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-oauth-config\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.026119 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.026022 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp6xl\" (UniqueName: \"kubernetes.io/projected/6ede2c99-11cb-4bf3-b82c-97831c28dbca-kube-api-access-mp6xl\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.026119 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.026098 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-oauth-serving-cert\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.026249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.026133 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-config\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.026249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.026151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-trusted-ca-bundle\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.071260 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.071224 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:00.127391 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127351 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-service-ca\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.127574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127399 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-serving-cert\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.127574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127428 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-oauth-config\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.127574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127449 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp6xl\" (UniqueName: \"kubernetes.io/projected/6ede2c99-11cb-4bf3-b82c-97831c28dbca-kube-api-access-mp6xl\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.127574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127478 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-oauth-serving-cert\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.127574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127501 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-config\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.127574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.127521 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-trusted-ca-bundle\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.128217 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.128188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-service-ca\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.128217 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.128213 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-oauth-serving-cert\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.128331 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.128198 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-config\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.128641 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.128619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-trusted-ca-bundle\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.129880 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.129853 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-serving-cert\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.129995 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.129891 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-oauth-config\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.137282 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.137261 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp6xl\" (UniqueName: \"kubernetes.io/projected/6ede2c99-11cb-4bf3-b82c-97831c28dbca-kube-api-access-mp6xl\") pod \"console-5f95df6849-jmlpc\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.228989 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.228895 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:00.360400 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.360362 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f95df6849-jmlpc"] Jul 14 19:49:00.364110 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:49:00.364083 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ede2c99_11cb_4bf3_b82c_97831c28dbca.slice/crio-0ef22e385083c1b913f2efb41b25d564ed10018e6fedf04ba812f1b9d2cb9fa3 WatchSource:0}: Error finding container 0ef22e385083c1b913f2efb41b25d564ed10018e6fedf04ba812f1b9d2cb9fa3: Status 404 returned error can't find the container with id 0ef22e385083c1b913f2efb41b25d564ed10018e6fedf04ba812f1b9d2cb9fa3 Jul 14 19:49:00.679896 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.679856 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95df6849-jmlpc" event={"ID":"6ede2c99-11cb-4bf3-b82c-97831c28dbca","Type":"ContainerStarted","Data":"9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702"} Jul 14 19:49:00.680084 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.679915 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95df6849-jmlpc" event={"ID":"6ede2c99-11cb-4bf3-b82c-97831c28dbca","Type":"ContainerStarted","Data":"0ef22e385083c1b913f2efb41b25d564ed10018e6fedf04ba812f1b9d2cb9fa3"} Jul 14 19:49:00.699205 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:00.699152 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f95df6849-jmlpc" podStartSLOduration=1.699134276 podStartE2EDuration="1.699134276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:49:00.696980398 +0000 UTC m=+146.145606853" watchObservedRunningTime="2026-07-14 19:49:00.699134276 +0000 UTC m=+146.147760777" Jul 14 19:49:10.229080 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:10.229034 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:10.229548 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:10.229110 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:10.233712 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:10.233686 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:10.718387 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:10.718352 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:49:10.986017 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:10.985934 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:12.275394 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:12.275364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:27.798513 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:27.798439 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:29.223297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:29.223265 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:45.776967 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:45.776932 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:45.837361 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:49:45.837327 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:15.916384 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:15.916352 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:16.019929 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.019900 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:16.706232 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.706196 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r"] Jul 14 19:50:16.709399 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.709381 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.712015 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.711990 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 19:50:16.712152 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.711997 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 19:50:16.712782 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.712767 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-728tr\"" Jul 14 19:50:16.722354 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.722331 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r"] Jul 14 19:50:16.757507 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.757475 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg77c\" (UniqueName: \"kubernetes.io/projected/7db93ea4-c517-41ef-9544-31b7cc165c76-kube-api-access-cg77c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.757507 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.757508 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.757693 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.757539 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.858166 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.858113 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg77c\" (UniqueName: \"kubernetes.io/projected/7db93ea4-c517-41ef-9544-31b7cc165c76-kube-api-access-cg77c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.858166 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.858173 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.858400 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.858217 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.858561 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.858545 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.858600 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.858578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:16.867514 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:16.867485 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg77c\" (UniqueName: \"kubernetes.io/projected/7db93ea4-c517-41ef-9544-31b7cc165c76-kube-api-access-cg77c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:17.018628 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:17.018544 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:17.144765 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:17.144738 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r"] Jul 14 19:50:17.147039 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:50:17.147016 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db93ea4_c517_41ef_9544_31b7cc165c76.slice/crio-53be225168a7b9ff7deb3a90d5a30c3e49b2aaa6f48ca7572e1d40e26d9a5074 WatchSource:0}: Error finding container 53be225168a7b9ff7deb3a90d5a30c3e49b2aaa6f48ca7572e1d40e26d9a5074: Status 404 returned error can't find the container with id 53be225168a7b9ff7deb3a90d5a30c3e49b2aaa6f48ca7572e1d40e26d9a5074 Jul 14 19:50:17.955299 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:17.955211 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" event={"ID":"7db93ea4-c517-41ef-9544-31b7cc165c76","Type":"ContainerStarted","Data":"53be225168a7b9ff7deb3a90d5a30c3e49b2aaa6f48ca7572e1d40e26d9a5074"} Jul 14 19:50:18.432099 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:18.432052 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:25.987412 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:25.987374 2553 generic.go:356] "Generic (PLEG): container finished" podID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerID="3374952c1b62f5b60a423916df9a755c78962b22eae614ee692e40818e130839" exitCode=0 Jul 14 19:50:25.987809 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:25.987436 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" event={"ID":"7db93ea4-c517-41ef-9544-31b7cc165c76","Type":"ContainerDied","Data":"3374952c1b62f5b60a423916df9a755c78962b22eae614ee692e40818e130839"} Jul 14 19:50:29.000472 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:29.000438 2553 generic.go:356] "Generic (PLEG): container finished" podID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerID="63432db2ee9b13cc3e0cf95b568a2808ad920308f604220ca8477e01f8dde572" exitCode=0 Jul 14 19:50:29.000842 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:29.000504 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" event={"ID":"7db93ea4-c517-41ef-9544-31b7cc165c76","Type":"ContainerDied","Data":"63432db2ee9b13cc3e0cf95b568a2808ad920308f604220ca8477e01f8dde572"} Jul 14 19:50:35.395747 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:35.395687 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:38.035783 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:38.035753 2553 generic.go:356] "Generic (PLEG): container finished" podID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerID="bfb294c1a17501c3a6a8afbc4e2b052699e0144ff8f1c20d49624356855892cc" exitCode=0 Jul 14 19:50:38.036158 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:38.035793 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" event={"ID":"7db93ea4-c517-41ef-9544-31b7cc165c76","Type":"ContainerDied","Data":"bfb294c1a17501c3a6a8afbc4e2b052699e0144ff8f1c20d49624356855892cc"} Jul 14 19:50:39.199621 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.199586 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:39.346858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.346767 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-bundle\") pod \"7db93ea4-c517-41ef-9544-31b7cc165c76\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " Jul 14 19:50:39.346858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.346803 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-util\") pod \"7db93ea4-c517-41ef-9544-31b7cc165c76\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " Jul 14 19:50:39.346858 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.346833 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg77c\" (UniqueName: \"kubernetes.io/projected/7db93ea4-c517-41ef-9544-31b7cc165c76-kube-api-access-cg77c\") pod \"7db93ea4-c517-41ef-9544-31b7cc165c76\" (UID: \"7db93ea4-c517-41ef-9544-31b7cc165c76\") " Jul 14 19:50:39.347457 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.347430 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-bundle" (OuterVolumeSpecName: "bundle") pod "7db93ea4-c517-41ef-9544-31b7cc165c76" (UID: "7db93ea4-c517-41ef-9544-31b7cc165c76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:50:39.348943 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.348919 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db93ea4-c517-41ef-9544-31b7cc165c76-kube-api-access-cg77c" (OuterVolumeSpecName: "kube-api-access-cg77c") pod "7db93ea4-c517-41ef-9544-31b7cc165c76" (UID: "7db93ea4-c517-41ef-9544-31b7cc165c76"). InnerVolumeSpecName "kube-api-access-cg77c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:50:39.351962 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.351935 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-util" (OuterVolumeSpecName: "util") pod "7db93ea4-c517-41ef-9544-31b7cc165c76" (UID: "7db93ea4-c517-41ef-9544-31b7cc165c76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:50:39.448003 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.447961 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:50:39.448003 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.447997 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7db93ea4-c517-41ef-9544-31b7cc165c76-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:50:39.448003 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:39.448007 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg77c\" (UniqueName: \"kubernetes.io/projected/7db93ea4-c517-41ef-9544-31b7cc165c76-kube-api-access-cg77c\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:50:40.045385 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:40.045357 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" Jul 14 19:50:40.045385 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:40.045361 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbc8r" event={"ID":"7db93ea4-c517-41ef-9544-31b7cc165c76","Type":"ContainerDied","Data":"53be225168a7b9ff7deb3a90d5a30c3e49b2aaa6f48ca7572e1d40e26d9a5074"} Jul 14 19:50:40.045593 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:40.045402 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53be225168a7b9ff7deb3a90d5a30c3e49b2aaa6f48ca7572e1d40e26d9a5074" Jul 14 19:50:42.786015 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:42.785972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:43.800220 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:43.800175 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:44.747295 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747259 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f"] Jul 14 19:50:44.747545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747533 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="extract" Jul 14 19:50:44.747586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747546 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="extract" Jul 14 19:50:44.747586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747555 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="util" Jul 14 19:50:44.747586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747560 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="util" Jul 14 19:50:44.747586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747571 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="pull" Jul 14 19:50:44.747586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747577 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="pull" Jul 14 19:50:44.747737 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.747618 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db93ea4-c517-41ef-9544-31b7cc165c76" containerName="extract" Jul 14 19:50:44.750775 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.750758 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:44.754449 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.754422 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ztvfd\"" Jul 14 19:50:44.754594 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.754498 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:50:44.754594 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.754497 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 19:50:44.764051 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.764025 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f"] Jul 14 19:50:44.892780 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.892749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91ccbf88-7d52-4917-8cd7-6f1d4ff23783-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nts4f\" (UID: \"91ccbf88-7d52-4917-8cd7-6f1d4ff23783\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:44.893218 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.892792 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrhk\" (UniqueName: \"kubernetes.io/projected/91ccbf88-7d52-4917-8cd7-6f1d4ff23783-kube-api-access-jqrhk\") pod \"cert-manager-operator-controller-manager-55c444bf86-nts4f\" (UID: \"91ccbf88-7d52-4917-8cd7-6f1d4ff23783\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:44.993209 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.993172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91ccbf88-7d52-4917-8cd7-6f1d4ff23783-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nts4f\" (UID: \"91ccbf88-7d52-4917-8cd7-6f1d4ff23783\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:44.993392 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.993232 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrhk\" (UniqueName: \"kubernetes.io/projected/91ccbf88-7d52-4917-8cd7-6f1d4ff23783-kube-api-access-jqrhk\") pod \"cert-manager-operator-controller-manager-55c444bf86-nts4f\" (UID: \"91ccbf88-7d52-4917-8cd7-6f1d4ff23783\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:44.993659 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:44.993635 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91ccbf88-7d52-4917-8cd7-6f1d4ff23783-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nts4f\" (UID: \"91ccbf88-7d52-4917-8cd7-6f1d4ff23783\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:45.004631 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:45.004574 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrhk\" (UniqueName: \"kubernetes.io/projected/91ccbf88-7d52-4917-8cd7-6f1d4ff23783-kube-api-access-jqrhk\") pod \"cert-manager-operator-controller-manager-55c444bf86-nts4f\" (UID: \"91ccbf88-7d52-4917-8cd7-6f1d4ff23783\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:45.059257 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:45.059222 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" Jul 14 19:50:45.191141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:45.191109 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f"] Jul 14 19:50:45.195603 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:50:45.195575 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ccbf88_7d52_4917_8cd7_6f1d4ff23783.slice/crio-a58f5f7e9535c5cd1efe906b09a9dd5a26814bee6731d02748c3bcbb78dbc2ed WatchSource:0}: Error finding container a58f5f7e9535c5cd1efe906b09a9dd5a26814bee6731d02748c3bcbb78dbc2ed: Status 404 returned error can't find the container with id a58f5f7e9535c5cd1efe906b09a9dd5a26814bee6731d02748c3bcbb78dbc2ed Jul 14 19:50:46.070236 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:46.070195 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" event={"ID":"91ccbf88-7d52-4917-8cd7-6f1d4ff23783","Type":"ContainerStarted","Data":"a58f5f7e9535c5cd1efe906b09a9dd5a26814bee6731d02748c3bcbb78dbc2ed"} Jul 14 19:50:46.451928 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:46.451887 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:46.552447 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:46.552402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:47.076651 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:47.076556 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" event={"ID":"91ccbf88-7d52-4917-8cd7-6f1d4ff23783","Type":"ContainerStarted","Data":"1571e1f3b681e08c7fc9904dc40be4c5961704bd969c9c5d7d36722eaf6cdd12"} Jul 14 19:50:47.111690 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:47.111627 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nts4f" podStartSLOduration=1.5248847909999999 podStartE2EDuration="3.111607933s" totalImagesPullingTime="1.586723142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:47.109096122 +0000 UTC m=+252.557722559" watchObservedRunningTime="2026-07-14 19:50:47.111607933 +0000 UTC m=+252.560234377" Jul 14 19:50:47.556423 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:47.556386 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:47.650282 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:47.650251 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:48.432597 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.432562 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889"] Jul 14 19:50:48.435527 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.435510 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.438425 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.438401 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 19:50:48.438610 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.438590 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-728tr\"" Jul 14 19:50:48.438661 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.438595 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 19:50:48.450525 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.450502 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889"] Jul 14 19:50:48.522836 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.522803 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.523006 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.522841 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbt7r\" (UniqueName: \"kubernetes.io/projected/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-kube-api-access-lbt7r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.523006 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.522917 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.585281 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.585248 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:48.623381 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.623343 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbt7r\" (UniqueName: \"kubernetes.io/projected/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-kube-api-access-lbt7r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.623553 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.623405 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.623553 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.623492 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.623833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.623816 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.623870 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.623848 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.645774 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.645746 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbt7r\" (UniqueName: \"kubernetes.io/projected/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-kube-api-access-lbt7r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.739917 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.739840 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:48.744561 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.744537 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:48.842881 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.842839 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j9485"] Jul 14 19:50:48.846814 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.846791 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:48.849423 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.849399 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 19:50:48.849668 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.849593 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wfknd\"" Jul 14 19:50:48.849668 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.849649 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 19:50:48.860978 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.860949 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j9485"] Jul 14 19:50:48.882539 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.882512 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889"] Jul 14 19:50:48.883836 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:50:48.883808 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37eb2f8_2cba_45a2_957e_e3edcc9fb026.slice/crio-2b7374a73841ff38ce5cbd806c8cbff2e5fefaaef4e2e5451de67a3e9ef9ead7 WatchSource:0}: Error finding container 2b7374a73841ff38ce5cbd806c8cbff2e5fefaaef4e2e5451de67a3e9ef9ead7: Status 404 returned error can't find the container with id 2b7374a73841ff38ce5cbd806c8cbff2e5fefaaef4e2e5451de67a3e9ef9ead7 Jul 14 19:50:48.925917 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.925878 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4hn\" (UniqueName: \"kubernetes.io/projected/b743dd3d-fb23-4c49-ba3f-6f703414dfbe-kube-api-access-ww4hn\") pod \"cert-manager-webhook-6486ffd997-j9485\" (UID: \"b743dd3d-fb23-4c49-ba3f-6f703414dfbe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:48.926093 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:48.925951 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b743dd3d-fb23-4c49-ba3f-6f703414dfbe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j9485\" (UID: \"b743dd3d-fb23-4c49-ba3f-6f703414dfbe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:49.027031 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.026989 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b743dd3d-fb23-4c49-ba3f-6f703414dfbe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j9485\" (UID: \"b743dd3d-fb23-4c49-ba3f-6f703414dfbe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:49.027247 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.027082 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4hn\" (UniqueName: \"kubernetes.io/projected/b743dd3d-fb23-4c49-ba3f-6f703414dfbe-kube-api-access-ww4hn\") pod \"cert-manager-webhook-6486ffd997-j9485\" (UID: \"b743dd3d-fb23-4c49-ba3f-6f703414dfbe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:49.054177 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.054130 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4hn\" (UniqueName: \"kubernetes.io/projected/b743dd3d-fb23-4c49-ba3f-6f703414dfbe-kube-api-access-ww4hn\") pod \"cert-manager-webhook-6486ffd997-j9485\" (UID: \"b743dd3d-fb23-4c49-ba3f-6f703414dfbe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:49.054490 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.054473 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b743dd3d-fb23-4c49-ba3f-6f703414dfbe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j9485\" (UID: \"b743dd3d-fb23-4c49-ba3f-6f703414dfbe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:49.086574 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.086543 2553 generic.go:356] "Generic (PLEG): container finished" podID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerID="67be8fd69a583d328fdb2f1355d930d9934171787a448c03733be29021873471" exitCode=0 Jul 14 19:50:49.086728 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.086587 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" event={"ID":"a37eb2f8-2cba-45a2-957e-e3edcc9fb026","Type":"ContainerDied","Data":"67be8fd69a583d328fdb2f1355d930d9934171787a448c03733be29021873471"} Jul 14 19:50:49.086728 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.086642 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" event={"ID":"a37eb2f8-2cba-45a2-957e-e3edcc9fb026","Type":"ContainerStarted","Data":"2b7374a73841ff38ce5cbd806c8cbff2e5fefaaef4e2e5451de67a3e9ef9ead7"} Jul 14 19:50:49.171287 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.171236 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:49.299419 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:49.299390 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j9485"] Jul 14 19:50:49.302307 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:50:49.302282 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb743dd3d_fb23_4c49_ba3f_6f703414dfbe.slice/crio-e3e5f6143b851e09087600b7583ed4d9e8b4449b59bdb060ccddb0aa477c4f2c WatchSource:0}: Error finding container e3e5f6143b851e09087600b7583ed4d9e8b4449b59bdb060ccddb0aa477c4f2c: Status 404 returned error can't find the container with id e3e5f6143b851e09087600b7583ed4d9e8b4449b59bdb060ccddb0aa477c4f2c Jul 14 19:50:50.092265 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:50.092186 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" event={"ID":"b743dd3d-fb23-4c49-ba3f-6f703414dfbe","Type":"ContainerStarted","Data":"e3e5f6143b851e09087600b7583ed4d9e8b4449b59bdb060ccddb0aa477c4f2c"} Jul 14 19:50:50.409845 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:50.409804 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:50.981042 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:50.980964 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:53.107567 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.107530 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" event={"ID":"b743dd3d-fb23-4c49-ba3f-6f703414dfbe","Type":"ContainerStarted","Data":"e26bb66ebaf70b47bc730004803a1274d11be96eca4e055e4b1b7ec8ecbd3dec"} Jul 14 19:50:53.108005 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.107578 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:53.109258 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.109232 2553 generic.go:356] "Generic (PLEG): container finished" podID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerID="0e8e65744bbc4afaf018161611dfb8924e51e90597f165098aeb5721de306974" exitCode=0 Jul 14 19:50:53.109335 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.109307 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" event={"ID":"a37eb2f8-2cba-45a2-957e-e3edcc9fb026","Type":"ContainerDied","Data":"0e8e65744bbc4afaf018161611dfb8924e51e90597f165098aeb5721de306974"} Jul 14 19:50:53.125857 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.125814 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" podStartSLOduration=2.11836777 podStartE2EDuration="5.125797337s" totalImagesPullingTime="3.007429567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:53.124644029 +0000 UTC m=+258.573270501" watchObservedRunningTime="2026-07-14 19:50:53.125797337 +0000 UTC m=+258.574423780" Jul 14 19:50:53.259832 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.259790 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:53.360122 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:53.360023 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:54.115571 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:54.115539 2553 generic.go:356] "Generic (PLEG): container finished" podID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerID="bc28cf0387155ceb8984f49ffe2c86b5f81ba43c38e66d0b622c2122e2971fa2" exitCode=0 Jul 14 19:50:54.116008 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:54.115631 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" event={"ID":"a37eb2f8-2cba-45a2-957e-e3edcc9fb026","Type":"ContainerDied","Data":"bc28cf0387155ceb8984f49ffe2c86b5f81ba43c38e66d0b622c2122e2971fa2"} Jul 14 19:50:55.243144 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.243119 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:55.385189 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.385157 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-util\") pod \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " Jul 14 19:50:55.385365 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.385224 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-bundle\") pod \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " Jul 14 19:50:55.385365 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.385256 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbt7r\" (UniqueName: \"kubernetes.io/projected/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-kube-api-access-lbt7r\") pod \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\" (UID: \"a37eb2f8-2cba-45a2-957e-e3edcc9fb026\") " Jul 14 19:50:55.385611 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.385584 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-bundle" (OuterVolumeSpecName: "bundle") pod "a37eb2f8-2cba-45a2-957e-e3edcc9fb026" (UID: "a37eb2f8-2cba-45a2-957e-e3edcc9fb026"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:50:55.387376 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.387352 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-kube-api-access-lbt7r" (OuterVolumeSpecName: "kube-api-access-lbt7r") pod "a37eb2f8-2cba-45a2-957e-e3edcc9fb026" (UID: "a37eb2f8-2cba-45a2-957e-e3edcc9fb026"). InnerVolumeSpecName "kube-api-access-lbt7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:50:55.389177 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.389152 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-util" (OuterVolumeSpecName: "util") pod "a37eb2f8-2cba-45a2-957e-e3edcc9fb026" (UID: "a37eb2f8-2cba-45a2-957e-e3edcc9fb026"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:50:55.486304 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.486271 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:50:55.486304 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.486299 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbt7r\" (UniqueName: \"kubernetes.io/projected/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-kube-api-access-lbt7r\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:50:55.486304 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.486309 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a37eb2f8-2cba-45a2-957e-e3edcc9fb026-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:50:55.842666 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.842590 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:55.939247 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:55.939213 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:56.125138 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:56.125044 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" Jul 14 19:50:56.125138 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:56.125058 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fws889" event={"ID":"a37eb2f8-2cba-45a2-957e-e3edcc9fb026","Type":"ContainerDied","Data":"2b7374a73841ff38ce5cbd806c8cbff2e5fefaaef4e2e5451de67a3e9ef9ead7"} Jul 14 19:50:56.125138 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:56.125127 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b7374a73841ff38ce5cbd806c8cbff2e5fefaaef4e2e5451de67a3e9ef9ead7" Jul 14 19:50:59.117990 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:59.117961 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j9485" Jul 14 19:50:59.790359 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:50:59.790319 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:04.594851 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:04.594819 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:06.623864 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.623830 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65"] Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624141 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="pull" Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624151 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="pull" Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624160 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="extract" Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624165 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="extract" Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624175 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="util" Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624180 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="util" Jul 14 19:51:06.624249 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.624232 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37eb2f8-2cba-45a2-957e-e3edcc9fb026" containerName="extract" Jul 14 19:51:06.666276 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.666240 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65"] Jul 14 19:51:06.666417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.666338 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.668999 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.668978 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 19:51:06.669913 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.669885 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:51:06.670027 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.669925 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2frcb\"" Jul 14 19:51:06.677738 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.677715 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2f6\" (UniqueName: \"kubernetes.io/projected/ce986d47-fd2f-4a8f-a9de-0438f1b6ad44-kube-api-access-mb2f6\") pod \"openshift-lws-operator-fd8ccff4c-dwk65\" (UID: \"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.677835 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.677756 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce986d47-fd2f-4a8f-a9de-0438f1b6ad44-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dwk65\" (UID: \"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.778726 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.778683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce986d47-fd2f-4a8f-a9de-0438f1b6ad44-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dwk65\" (UID: \"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.778923 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.778774 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2f6\" (UniqueName: \"kubernetes.io/projected/ce986d47-fd2f-4a8f-a9de-0438f1b6ad44-kube-api-access-mb2f6\") pod \"openshift-lws-operator-fd8ccff4c-dwk65\" (UID: \"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.779104 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.779084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce986d47-fd2f-4a8f-a9de-0438f1b6ad44-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dwk65\" (UID: \"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.789586 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.789560 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2f6\" (UniqueName: \"kubernetes.io/projected/ce986d47-fd2f-4a8f-a9de-0438f1b6ad44-kube-api-access-mb2f6\") pod \"openshift-lws-operator-fd8ccff4c-dwk65\" (UID: \"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:06.975148 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:06.975106 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" Jul 14 19:51:07.130757 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:07.130726 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65"] Jul 14 19:51:07.134324 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:51:07.134293 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce986d47_fd2f_4a8f_a9de_0438f1b6ad44.slice/crio-c38a2ae735bd90a218bd2ff46940bc565b8469a3c6f1a58fe646d70370cafbcf WatchSource:0}: Error finding container c38a2ae735bd90a218bd2ff46940bc565b8469a3c6f1a58fe646d70370cafbcf: Status 404 returned error can't find the container with id c38a2ae735bd90a218bd2ff46940bc565b8469a3c6f1a58fe646d70370cafbcf Jul 14 19:51:07.172910 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:07.172867 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" event={"ID":"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44","Type":"ContainerStarted","Data":"c38a2ae735bd90a218bd2ff46940bc565b8469a3c6f1a58fe646d70370cafbcf"} Jul 14 19:51:09.184689 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:09.184595 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" event={"ID":"ce986d47-fd2f-4a8f-a9de-0438f1b6ad44","Type":"ContainerStarted","Data":"8b17daa87359a6b11e9632b036e33bf9a322f65ad1a3804062f83402ba70369a"} Jul 14 19:51:09.215441 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:09.215392 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dwk65" podStartSLOduration=1.427563406 podStartE2EDuration="3.215376289s" totalImagesPullingTime="1.787812883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:09.214271502 +0000 UTC m=+274.662897940" watchObservedRunningTime="2026-07-14 19:51:09.215376289 +0000 UTC m=+274.664002733" Jul 14 19:51:11.250366 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:11.250333 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:11.965190 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:11.965161 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:14.479749 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.479719 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g"] Jul 14 19:51:14.481897 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.481880 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.484847 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.484823 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 19:51:14.484962 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.484824 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 19:51:14.485801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.485785 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-728tr\"" Jul 14 19:51:14.495428 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.495403 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g"] Jul 14 19:51:14.549545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.549503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxrf\" (UniqueName: \"kubernetes.io/projected/b22dd831-fcc0-4369-9f27-f196f8c9399d-kube-api-access-kvxrf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.549720 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.549623 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.549720 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.549653 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.650558 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.650521 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.650558 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.650562 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.650752 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.650605 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxrf\" (UniqueName: \"kubernetes.io/projected/b22dd831-fcc0-4369-9f27-f196f8c9399d-kube-api-access-kvxrf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.650994 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.650972 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.651029 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.650991 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.661800 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.661777 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxrf\" (UniqueName: \"kubernetes.io/projected/b22dd831-fcc0-4369-9f27-f196f8c9399d-kube-api-access-kvxrf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.790768 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.790661 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:14.919171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:14.919142 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g"] Jul 14 19:51:14.921738 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:51:14.921714 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22dd831_fcc0_4369_9f27_f196f8c9399d.slice/crio-65b1e8f9a935de562dbad32b4c6226638d6ba743182c639d2fe94c6f8c360304 WatchSource:0}: Error finding container 65b1e8f9a935de562dbad32b4c6226638d6ba743182c639d2fe94c6f8c360304: Status 404 returned error can't find the container with id 65b1e8f9a935de562dbad32b4c6226638d6ba743182c639d2fe94c6f8c360304 Jul 14 19:51:15.210501 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:15.210469 2553 generic.go:356] "Generic (PLEG): container finished" podID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerID="ccd6e63471c2d596b2370912e7766f8c3746491c0d5d386f01c714a1f3e6c1aa" exitCode=0 Jul 14 19:51:15.210685 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:15.210536 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" event={"ID":"b22dd831-fcc0-4369-9f27-f196f8c9399d","Type":"ContainerDied","Data":"ccd6e63471c2d596b2370912e7766f8c3746491c0d5d386f01c714a1f3e6c1aa"} Jul 14 19:51:15.210685 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:15.210585 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" event={"ID":"b22dd831-fcc0-4369-9f27-f196f8c9399d","Type":"ContainerStarted","Data":"65b1e8f9a935de562dbad32b4c6226638d6ba743182c639d2fe94c6f8c360304"} Jul 14 19:51:16.216189 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:16.216154 2553 generic.go:356] "Generic (PLEG): container finished" podID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerID="b899f1efa4307fe606e1128552237989b0878e919f8fd06e7d277c27d292a7b5" exitCode=0 Jul 14 19:51:16.216566 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:16.216232 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" event={"ID":"b22dd831-fcc0-4369-9f27-f196f8c9399d","Type":"ContainerDied","Data":"b899f1efa4307fe606e1128552237989b0878e919f8fd06e7d277c27d292a7b5"} Jul 14 19:51:17.221878 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:17.221842 2553 generic.go:356] "Generic (PLEG): container finished" podID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerID="93b818dafae2768b8c2548993b1d659a85d8b5851890ceb9bb803d524dd12c50" exitCode=0 Jul 14 19:51:17.222263 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:17.221964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" event={"ID":"b22dd831-fcc0-4369-9f27-f196f8c9399d","Type":"ContainerDied","Data":"93b818dafae2768b8c2548993b1d659a85d8b5851890ceb9bb803d524dd12c50"} Jul 14 19:51:18.346857 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.346835 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:18.382445 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.382412 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-util\") pod \"b22dd831-fcc0-4369-9f27-f196f8c9399d\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " Jul 14 19:51:18.382621 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.382459 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvxrf\" (UniqueName: \"kubernetes.io/projected/b22dd831-fcc0-4369-9f27-f196f8c9399d-kube-api-access-kvxrf\") pod \"b22dd831-fcc0-4369-9f27-f196f8c9399d\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " Jul 14 19:51:18.382621 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.382497 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-bundle\") pod \"b22dd831-fcc0-4369-9f27-f196f8c9399d\" (UID: \"b22dd831-fcc0-4369-9f27-f196f8c9399d\") " Jul 14 19:51:18.383398 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.383366 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-bundle" (OuterVolumeSpecName: "bundle") pod "b22dd831-fcc0-4369-9f27-f196f8c9399d" (UID: "b22dd831-fcc0-4369-9f27-f196f8c9399d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:18.384616 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.384593 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22dd831-fcc0-4369-9f27-f196f8c9399d-kube-api-access-kvxrf" (OuterVolumeSpecName: "kube-api-access-kvxrf") pod "b22dd831-fcc0-4369-9f27-f196f8c9399d" (UID: "b22dd831-fcc0-4369-9f27-f196f8c9399d"). InnerVolumeSpecName "kube-api-access-kvxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:18.387910 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.387883 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-util" (OuterVolumeSpecName: "util") pod "b22dd831-fcc0-4369-9f27-f196f8c9399d" (UID: "b22dd831-fcc0-4369-9f27-f196f8c9399d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:18.483341 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.483258 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:51:18.483341 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.483285 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxrf\" (UniqueName: \"kubernetes.io/projected/b22dd831-fcc0-4369-9f27-f196f8c9399d-kube-api-access-kvxrf\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:51:18.483341 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:18.483295 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b22dd831-fcc0-4369-9f27-f196f8c9399d-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:51:19.232491 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:19.232462 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" Jul 14 19:51:19.232650 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:19.232457 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9786g" event={"ID":"b22dd831-fcc0-4369-9f27-f196f8c9399d","Type":"ContainerDied","Data":"65b1e8f9a935de562dbad32b4c6226638d6ba743182c639d2fe94c6f8c360304"} Jul 14 19:51:19.232650 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:19.232575 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b1e8f9a935de562dbad32b4c6226638d6ba743182c639d2fe94c6f8c360304" Jul 14 19:51:25.429824 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:25.429786 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:25.525047 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:25.525013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:26.171447 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.171412 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:26.274101 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.274055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:26.377695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.377661 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x"] Jul 14 19:51:26.377859 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.377779 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:26.378009 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.377996 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="extract" Jul 14 19:51:26.378044 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.378011 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="extract" Jul 14 19:51:26.378044 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.378033 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="util" Jul 14 19:51:26.378044 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.378038 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="util" Jul 14 19:51:26.378158 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.378045 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="pull" Jul 14 19:51:26.378158 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.378050 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="pull" Jul 14 19:51:26.378158 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.378113 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b22dd831-fcc0-4369-9f27-f196f8c9399d" containerName="extract" Jul 14 19:51:26.380980 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.380965 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.384519 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.384497 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 19:51:26.384658 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.384521 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 19:51:26.384658 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.384605 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 19:51:26.384954 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.384941 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5l9cj\"" Jul 14 19:51:26.385002 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.384949 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 19:51:26.403609 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.403576 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x"] Jul 14 19:51:26.452835 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.452749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8ef32cb-425c-4973-9e5b-18a520e9177e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.453202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.452853 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8ef32cb-425c-4973-9e5b-18a520e9177e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.453202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.452877 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8ef32cb-425c-4973-9e5b-18a520e9177e-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.453202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.452927 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8ef32cb-425c-4973-9e5b-18a520e9177e-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.453202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.452972 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjdc\" (UniqueName: \"kubernetes.io/projected/e8ef32cb-425c-4973-9e5b-18a520e9177e-kube-api-access-ljjdc\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.549136 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.549109 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:26.553972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.553952 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8ef32cb-425c-4973-9e5b-18a520e9177e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.554033 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.553983 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8ef32cb-425c-4973-9e5b-18a520e9177e-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.554033 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.554015 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8ef32cb-425c-4973-9e5b-18a520e9177e-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.554122 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.554052 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjdc\" (UniqueName: \"kubernetes.io/projected/e8ef32cb-425c-4973-9e5b-18a520e9177e-kube-api-access-ljjdc\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.554122 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.554103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8ef32cb-425c-4973-9e5b-18a520e9177e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.554409 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.554391 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8ef32cb-425c-4973-9e5b-18a520e9177e-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.556326 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.556304 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8ef32cb-425c-4973-9e5b-18a520e9177e-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.556470 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.556453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8ef32cb-425c-4973-9e5b-18a520e9177e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.556507 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.556474 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8ef32cb-425c-4973-9e5b-18a520e9177e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.563658 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.563633 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjdc\" (UniqueName: \"kubernetes.io/projected/e8ef32cb-425c-4973-9e5b-18a520e9177e-kube-api-access-ljjdc\") pod \"opendatahub-operator-controller-manager-7f59db9768-pl58x\" (UID: \"e8ef32cb-425c-4973-9e5b-18a520e9177e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.652682 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.652654 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:26.690773 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.690741 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:26.855885 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:26.855857 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x"] Jul 14 19:51:26.858506 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:51:26.858478 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ef32cb_425c_4973_9e5b_18a520e9177e.slice/crio-69b584633587fdaf5eb09ba60fa4dea94775f4ac61eb9b3b1517cc46b1e4f59a WatchSource:0}: Error finding container 69b584633587fdaf5eb09ba60fa4dea94775f4ac61eb9b3b1517cc46b1e4f59a: Status 404 returned error can't find the container with id 69b584633587fdaf5eb09ba60fa4dea94775f4ac61eb9b3b1517cc46b1e4f59a Jul 14 19:51:27.268339 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:27.268308 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" event={"ID":"e8ef32cb-425c-4973-9e5b-18a520e9177e","Type":"ContainerStarted","Data":"69b584633587fdaf5eb09ba60fa4dea94775f4ac61eb9b3b1517cc46b1e4f59a"} Jul 14 19:51:28.632629 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:28.632597 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:28.733214 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:28.733186 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:32.292597 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:32.292556 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" event={"ID":"e8ef32cb-425c-4973-9e5b-18a520e9177e","Type":"ContainerStarted","Data":"e4fb10ee0df13ad556adf4c647fc0e8065e1331abf85757c3f5fb656be24ed53"} Jul 14 19:51:33.298594 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:33.298549 2553 generic.go:356] "Generic (PLEG): container finished" podID="e8ef32cb-425c-4973-9e5b-18a520e9177e" containerID="e4fb10ee0df13ad556adf4c647fc0e8065e1331abf85757c3f5fb656be24ed53" exitCode=0 Jul 14 19:51:33.299084 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:33.298599 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" event={"ID":"e8ef32cb-425c-4973-9e5b-18a520e9177e","Type":"ContainerDied","Data":"e4fb10ee0df13ad556adf4c647fc0e8065e1331abf85757c3f5fb656be24ed53"} Jul 14 19:51:34.305499 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:34.305462 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" event={"ID":"e8ef32cb-425c-4973-9e5b-18a520e9177e","Type":"ContainerStarted","Data":"a086a95199391eee97213c21507bd55ed879fb8474968ccbf5440b004a51c17b"} Jul 14 19:51:34.305951 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:34.305561 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:34.332158 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:34.332103 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" podStartSLOduration=2.040598387 podStartE2EDuration="8.33208716s" totalImagesPullingTime="5.26754299s" totalInitContainerRuntime="1.023945783s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:34.330782249 +0000 UTC m=+299.779408713" watchObservedRunningTime="2026-07-14 19:51:34.33208716 +0000 UTC m=+299.780713601" Jul 14 19:51:34.886983 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:34.886957 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:51:34.887456 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:34.887434 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:51:34.893844 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:34.893825 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 19:51:37.575384 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:37.575342 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:37.669838 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:37.669797 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:38.263188 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:38.263141 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:38.362091 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:38.362021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:44.502647 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.502615 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp"] Jul 14 19:51:44.507038 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.507015 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.510548 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.510527 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 19:51:44.510548 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.510547 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 19:51:44.510807 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.510794 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 19:51:44.510859 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.510791 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fh6nk\"" Jul 14 19:51:44.520796 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.520773 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp"] Jul 14 19:51:44.611946 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.611906 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-metrics-cert\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.612157 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.611962 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-manager-config\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.612157 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.612034 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-cert\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.612157 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.612111 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhkt\" (UniqueName: \"kubernetes.io/projected/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-kube-api-access-cmhkt\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.713290 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.713234 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-metrics-cert\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.713494 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.713306 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-manager-config\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.713494 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.713332 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-cert\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.713494 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.713371 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhkt\" (UniqueName: \"kubernetes.io/projected/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-kube-api-access-cmhkt\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.713979 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.713956 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-manager-config\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.715707 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.715680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-cert\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.715843 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.715823 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-metrics-cert\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.722784 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.722750 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhkt\" (UniqueName: \"kubernetes.io/projected/f9dd66bb-d2ee-4b0a-8124-79988bbfaf24-kube-api-access-cmhkt\") pod \"lws-controller-manager-5dc78d48f4-skzmp\" (UID: \"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.816266 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.816167 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:44.944367 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.944341 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp"] Jul 14 19:51:44.946909 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:51:44.946882 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dd66bb_d2ee_4b0a_8124_79988bbfaf24.slice/crio-8d5532420090f214736f9b4f7620802d8e7fc659bda73789bcbce7586cfe273b WatchSource:0}: Error finding container 8d5532420090f214736f9b4f7620802d8e7fc659bda73789bcbce7586cfe273b: Status 404 returned error can't find the container with id 8d5532420090f214736f9b4f7620802d8e7fc659bda73789bcbce7586cfe273b Jul 14 19:51:44.948937 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:44.948922 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:51:45.313370 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:45.313342 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-pl58x" Jul 14 19:51:45.357297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:45.357247 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" event={"ID":"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24","Type":"ContainerStarted","Data":"8d5532420090f214736f9b4f7620802d8e7fc659bda73789bcbce7586cfe273b"} Jul 14 19:51:46.953076 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:46.953033 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:47.079795 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:47.079762 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:48.054701 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:48.054665 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:48.122046 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:48.122014 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:48.223524 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:48.223491 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:48.322763 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:48.322670 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:48.591298 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:48.591219 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:48.728283 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:48.728246 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:49.030038 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:49.030010 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:50.381591 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:50.381551 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" event={"ID":"f9dd66bb-d2ee-4b0a-8124-79988bbfaf24","Type":"ContainerStarted","Data":"9fe5e2c0926f2c1a356120c0e746a92f8f28723d9ca78ed4aa8235704b3b4bec"} Jul 14 19:51:50.381999 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:50.381612 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:51:50.400049 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:50.399999 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" podStartSLOduration=1.9039221560000001 podStartE2EDuration="6.399985543s" totalImagesPullingTime="4.496063387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:50.399195817 +0000 UTC m=+315.847822263" watchObservedRunningTime="2026-07-14 19:51:50.399985543 +0000 UTC m=+315.848611987" Jul 14 19:51:57.378516 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:57.378480 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:57.468648 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:57.468610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:57.868771 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:57.868690 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:58.161556 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:58.161527 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:58.272347 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:58.272310 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:58.550402 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:58.550319 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:58.651270 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:58.651242 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:58.751705 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:51:58.751677 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:00.206603 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.206571 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:00.214855 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.214830 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd"] Jul 14 19:52:00.218048 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.218033 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.220681 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.220661 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 19:52:00.220813 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.220700 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 19:52:00.220950 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.220913 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 19:52:00.221124 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.221109 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 19:52:00.221195 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.221139 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 19:52:00.221195 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.221164 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8vpcp\"" Jul 14 19:52:00.221302 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.221139 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 19:52:00.231104 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.231084 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd"] Jul 14 19:52:00.237671 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237650 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.237762 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237685 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59228292-6033-4402-9935-3b486085902d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.237762 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237717 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59228292-6033-4402-9935-3b486085902d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.237830 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237762 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.237830 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237785 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j477h\" (UniqueName: \"kubernetes.io/projected/59228292-6033-4402-9935-3b486085902d-kube-api-access-j477h\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.237898 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237825 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59228292-6033-4402-9935-3b486085902d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.237898 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.237854 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.282642 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.282608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:00.338335 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338296 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.338335 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338338 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j477h\" (UniqueName: \"kubernetes.io/projected/59228292-6033-4402-9935-3b486085902d-kube-api-access-j477h\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.338545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59228292-6033-4402-9935-3b486085902d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.338545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338485 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.338545 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.338695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338569 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59228292-6033-4402-9935-3b486085902d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.338695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.338596 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59228292-6033-4402-9935-3b486085902d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.339173 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.339151 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59228292-6033-4402-9935-3b486085902d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.340637 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.340620 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.340805 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.340786 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.341005 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.340989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59228292-6033-4402-9935-3b486085902d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.341140 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.341035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59228292-6033-4402-9935-3b486085902d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.346906 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.346883 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59228292-6033-4402-9935-3b486085902d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.347296 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.347279 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j477h\" (UniqueName: \"kubernetes.io/projected/59228292-6033-4402-9935-3b486085902d-kube-api-access-j477h\") pod \"istiod-openshift-gateway-94bb8fbfd-2jzhd\" (UID: \"59228292-6033-4402-9935-3b486085902d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.527642 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.527545 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:00.660476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:00.660451 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd"] Jul 14 19:52:00.662303 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:52:00.662274 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59228292_6033_4402_9935_3b486085902d.slice/crio-d937fc54e0b1eab08502ff4662b026e5dcdf5a60c46a22af1c48dfb7fd4ab578 WatchSource:0}: Error finding container d937fc54e0b1eab08502ff4662b026e5dcdf5a60c46a22af1c48dfb7fd4ab578: Status 404 returned error can't find the container with id d937fc54e0b1eab08502ff4662b026e5dcdf5a60c46a22af1c48dfb7fd4ab578 Jul 14 19:52:01.332196 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:01.332155 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:01.388723 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:01.388698 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-skzmp" Jul 14 19:52:01.438622 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:01.438586 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" event={"ID":"59228292-6033-4402-9935-3b486085902d","Type":"ContainerStarted","Data":"d937fc54e0b1eab08502ff4662b026e5dcdf5a60c46a22af1c48dfb7fd4ab578"} Jul 14 19:52:01.561116 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:01.561055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:01.746092 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:01.746044 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:03.563432 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:03.563397 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:03.563682 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:03.563463 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:04.087906 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.087856 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:04.201029 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.200997 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:04.455195 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.455155 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" event={"ID":"59228292-6033-4402-9935-3b486085902d","Type":"ContainerStarted","Data":"f2161de9aa0e4e61ee562167159405d8e97009697f904f0c13f644882d22ada6"} Jul 14 19:52:04.455383 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.455367 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:04.457096 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.457049 2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-2jzhd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:52:04.457240 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.457136 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" podUID="59228292-6033-4402-9935-3b486085902d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:52:04.480235 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.480186 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" podStartSLOduration=1.581222578 podStartE2EDuration="4.480171985s" totalImagesPullingTime="2.898949407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:04.478819463 +0000 UTC m=+329.927445907" watchObservedRunningTime="2026-07-14 19:52:04.480171985 +0000 UTC m=+329.928798430" Jul 14 19:52:04.777758 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:04.777679 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:05.243551 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:05.243515 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:05.344720 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:05.344686 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:05.460707 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:05.460678 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2jzhd" Jul 14 19:52:48.332509 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:48.332428 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:48.431888 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:48.431853 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:48.532404 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:48.532376 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:59.170025 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:59.169990 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:59.231310 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:59.231266 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:59.305411 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:59.305379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:59.403294 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:59.403264 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:59.898259 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:52:59.898224 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:00.766072 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:00.766043 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:03.653347 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.653316 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm"] Jul 14 19:53:03.655797 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.655780 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.658208 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.658176 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:53:03.658208 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.658177 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:53:03.658368 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.658213 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slfsg\"" Jul 14 19:53:03.664337 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.664310 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm"] Jul 14 19:53:03.770514 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.770472 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5s8\" (UniqueName: \"kubernetes.io/projected/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-kube-api-access-vp5s8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.770514 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.770513 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.770720 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.770606 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.871795 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.871763 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5s8\" (UniqueName: \"kubernetes.io/projected/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-kube-api-access-vp5s8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.871795 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.871796 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.871981 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.871831 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.872164 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.872149 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.872212 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.872188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.881278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.881256 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5s8\" (UniqueName: \"kubernetes.io/projected/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-kube-api-access-vp5s8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:03.965269 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:03.965184 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:04.094044 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.094018 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm"] Jul 14 19:53:04.096612 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:53:04.096586 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e21b8f7_d1b5_47c8_91d0_d3cf4b6146ab.slice/crio-9a50cae421d109813e19f2a0e8416e1f456ea15d972aeaf5272616c94e6d5cb7 WatchSource:0}: Error finding container 9a50cae421d109813e19f2a0e8416e1f456ea15d972aeaf5272616c94e6d5cb7: Status 404 returned error can't find the container with id 9a50cae421d109813e19f2a0e8416e1f456ea15d972aeaf5272616c94e6d5cb7 Jul 14 19:53:04.452076 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.452021 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484"] Jul 14 19:53:04.454362 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.454345 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.464359 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.464335 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484"] Jul 14 19:53:04.477607 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.477574 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.477775 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.477633 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjs9\" (UniqueName: \"kubernetes.io/projected/73c6013e-2f06-43bf-ab99-e3e14b27ea62-kube-api-access-kvjs9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.477775 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.477749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.578939 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.578901 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.579185 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.578963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjs9\" (UniqueName: \"kubernetes.io/projected/73c6013e-2f06-43bf-ab99-e3e14b27ea62-kube-api-access-kvjs9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.579185 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.579008 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.579311 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.579292 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.579351 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.579323 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.588371 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.588351 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjs9\" (UniqueName: \"kubernetes.io/projected/73c6013e-2f06-43bf-ab99-e3e14b27ea62-kube-api-access-kvjs9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.727639 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.727540 2553 generic.go:356] "Generic (PLEG): container finished" podID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerID="c6685f57a461c1bb798020733d22796997a91d06e48e07f17f5ecfe41e6f2ba3" exitCode=0 Jul 14 19:53:04.728042 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.727634 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" event={"ID":"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab","Type":"ContainerDied","Data":"c6685f57a461c1bb798020733d22796997a91d06e48e07f17f5ecfe41e6f2ba3"} Jul 14 19:53:04.728042 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.727689 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" event={"ID":"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab","Type":"ContainerStarted","Data":"9a50cae421d109813e19f2a0e8416e1f456ea15d972aeaf5272616c94e6d5cb7"} Jul 14 19:53:04.762959 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.762926 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:04.853345 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.853315 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk"] Jul 14 19:53:04.855913 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.855883 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.866849 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.866818 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk"] Jul 14 19:53:04.881830 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.881801 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.881969 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.881950 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.882010 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.881983 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmpp8\" (UniqueName: \"kubernetes.io/projected/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-kube-api-access-nmpp8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.893024 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.892956 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484"] Jul 14 19:53:04.895082 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:53:04.895042 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c6013e_2f06_43bf_ab99_e3e14b27ea62.slice/crio-507c6405a2008927d84685f85faa9b75df406d3ce1a99df2b7e7dd56a78e7d6e WatchSource:0}: Error finding container 507c6405a2008927d84685f85faa9b75df406d3ce1a99df2b7e7dd56a78e7d6e: Status 404 returned error can't find the container with id 507c6405a2008927d84685f85faa9b75df406d3ce1a99df2b7e7dd56a78e7d6e Jul 14 19:53:04.982453 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.982428 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.982567 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.982464 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmpp8\" (UniqueName: \"kubernetes.io/projected/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-kube-api-access-nmpp8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.982567 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.982490 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.982883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.982859 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.982883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.982874 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:04.991916 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:04.991893 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmpp8\" (UniqueName: \"kubernetes.io/projected/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-kube-api-access-nmpp8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:05.168692 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.168657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:05.263544 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.263517 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm"] Jul 14 19:53:05.266114 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.266092 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.277813 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.277782 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm"] Jul 14 19:53:05.299877 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.299854 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk"] Jul 14 19:53:05.302009 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:53:05.301985 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3e7d92_e7c4_4cc3_9435_1328fc7db435.slice/crio-350570a32b9ba7a45c66a75545fad4499e2b33b08b17e6f8a4497f4c48008fec WatchSource:0}: Error finding container 350570a32b9ba7a45c66a75545fad4499e2b33b08b17e6f8a4497f4c48008fec: Status 404 returned error can't find the container with id 350570a32b9ba7a45c66a75545fad4499e2b33b08b17e6f8a4497f4c48008fec Jul 14 19:53:05.386749 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.386718 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.386857 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.386785 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqkv\" (UniqueName: \"kubernetes.io/projected/a8b7f543-5c73-4adb-b711-13c87679d19c-kube-api-access-4nqkv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.386857 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.386818 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.488084 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.488030 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqkv\" (UniqueName: \"kubernetes.io/projected/a8b7f543-5c73-4adb-b711-13c87679d19c-kube-api-access-4nqkv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.488245 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.488111 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.488245 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.488219 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.488505 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.488487 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.488556 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.488520 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.497805 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.497783 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqkv\" (UniqueName: \"kubernetes.io/projected/a8b7f543-5c73-4adb-b711-13c87679d19c-kube-api-access-4nqkv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.576906 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.576820 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:05.707143 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.707120 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm"] Jul 14 19:53:05.708934 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:53:05.708907 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b7f543_5c73_4adb_b711_13c87679d19c.slice/crio-0e72bedf487fc8ed9ed891ae5e6a0c200723aafe81256c44c0d8dc66e6164750 WatchSource:0}: Error finding container 0e72bedf487fc8ed9ed891ae5e6a0c200723aafe81256c44c0d8dc66e6164750: Status 404 returned error can't find the container with id 0e72bedf487fc8ed9ed891ae5e6a0c200723aafe81256c44c0d8dc66e6164750 Jul 14 19:53:05.739286 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.739252 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" event={"ID":"a8b7f543-5c73-4adb-b711-13c87679d19c","Type":"ContainerStarted","Data":"0e72bedf487fc8ed9ed891ae5e6a0c200723aafe81256c44c0d8dc66e6164750"} Jul 14 19:53:05.740896 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.740869 2553 generic.go:356] "Generic (PLEG): container finished" podID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerID="d8710a43c1933c278e18cbd290b396ae94b126167e9cc8418b13f786e4db326d" exitCode=0 Jul 14 19:53:05.741032 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.740927 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" event={"ID":"73c6013e-2f06-43bf-ab99-e3e14b27ea62","Type":"ContainerDied","Data":"d8710a43c1933c278e18cbd290b396ae94b126167e9cc8418b13f786e4db326d"} Jul 14 19:53:05.741032 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.740978 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" event={"ID":"73c6013e-2f06-43bf-ab99-e3e14b27ea62","Type":"ContainerStarted","Data":"507c6405a2008927d84685f85faa9b75df406d3ce1a99df2b7e7dd56a78e7d6e"} Jul 14 19:53:05.742889 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.742862 2553 generic.go:356] "Generic (PLEG): container finished" podID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerID="52ca74baf7e0f47d024eea17265952c6efc48affd5dde8759a09cf9e48236892" exitCode=0 Jul 14 19:53:05.743086 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.742949 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" event={"ID":"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab","Type":"ContainerDied","Data":"52ca74baf7e0f47d024eea17265952c6efc48affd5dde8759a09cf9e48236892"} Jul 14 19:53:05.744730 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.744710 2553 generic.go:356] "Generic (PLEG): container finished" podID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerID="905b833f125e0b3e91414591d1a3025aad4743835b7bd827079e17de404bfa20" exitCode=0 Jul 14 19:53:05.744837 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.744775 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" event={"ID":"1e3e7d92-e7c4-4cc3-9435-1328fc7db435","Type":"ContainerDied","Data":"905b833f125e0b3e91414591d1a3025aad4743835b7bd827079e17de404bfa20"} Jul 14 19:53:05.744837 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:05.744804 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" event={"ID":"1e3e7d92-e7c4-4cc3-9435-1328fc7db435","Type":"ContainerStarted","Data":"350570a32b9ba7a45c66a75545fad4499e2b33b08b17e6f8a4497f4c48008fec"} Jul 14 19:53:05.881698 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:53:05.881661 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e21b8f7_d1b5_47c8_91d0_d3cf4b6146ab.slice/crio-conmon-5ba1c66a37025ec50237e457d6602165ec2292f126bfc5f55d14fa2f9eb93654.scope\": RecentStats: unable to find data in memory cache]" Jul 14 19:53:06.751755 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:06.751679 2553 generic.go:356] "Generic (PLEG): container finished" podID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerID="7634f52c19f3e9b53eb76e2a26cd4b4e43b865cf461657565016b3c0ff4eb5ea" exitCode=0 Jul 14 19:53:06.752153 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:06.751753 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" event={"ID":"a8b7f543-5c73-4adb-b711-13c87679d19c","Type":"ContainerDied","Data":"7634f52c19f3e9b53eb76e2a26cd4b4e43b865cf461657565016b3c0ff4eb5ea"} Jul 14 19:53:06.754233 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:06.754197 2553 generic.go:356] "Generic (PLEG): container finished" podID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerID="5ba1c66a37025ec50237e457d6602165ec2292f126bfc5f55d14fa2f9eb93654" exitCode=0 Jul 14 19:53:06.754327 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:06.754285 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" event={"ID":"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab","Type":"ContainerDied","Data":"5ba1c66a37025ec50237e457d6602165ec2292f126bfc5f55d14fa2f9eb93654"} Jul 14 19:53:06.756195 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:06.756152 2553 generic.go:356] "Generic (PLEG): container finished" podID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerID="94bfd0aa95eaf38d321b41b179d165f7d2c27b91b6f1b5b7474b232b708c1c23" exitCode=0 Jul 14 19:53:06.756305 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:06.756227 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" event={"ID":"1e3e7d92-e7c4-4cc3-9435-1328fc7db435","Type":"ContainerDied","Data":"94bfd0aa95eaf38d321b41b179d165f7d2c27b91b6f1b5b7474b232b708c1c23"} Jul 14 19:53:07.766053 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.766021 2553 generic.go:356] "Generic (PLEG): container finished" podID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerID="2d194d2855101087d397419f29a42570ee114120b3bb0f2468097526c3b8092c" exitCode=0 Jul 14 19:53:07.766477 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.766096 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" event={"ID":"1e3e7d92-e7c4-4cc3-9435-1328fc7db435","Type":"ContainerDied","Data":"2d194d2855101087d397419f29a42570ee114120b3bb0f2468097526c3b8092c"} Jul 14 19:53:07.767925 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.767899 2553 generic.go:356] "Generic (PLEG): container finished" podID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerID="2a2ed14d0bd32e6745acc95493e3d6ddf166fe3c70ed6900f9dfda256df23f07" exitCode=0 Jul 14 19:53:07.768075 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.767980 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" event={"ID":"a8b7f543-5c73-4adb-b711-13c87679d19c","Type":"ContainerDied","Data":"2a2ed14d0bd32e6745acc95493e3d6ddf166fe3c70ed6900f9dfda256df23f07"} Jul 14 19:53:07.769775 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.769756 2553 generic.go:356] "Generic (PLEG): container finished" podID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerID="0ce6b1cdbdd85e4b553345490fea253b64c46b07d837d4a1bf29593aea5e12de" exitCode=0 Jul 14 19:53:07.769873 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.769829 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" event={"ID":"73c6013e-2f06-43bf-ab99-e3e14b27ea62","Type":"ContainerDied","Data":"0ce6b1cdbdd85e4b553345490fea253b64c46b07d837d4a1bf29593aea5e12de"} Jul 14 19:53:07.900000 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:07.899970 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:08.010613 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.010579 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-bundle\") pod \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " Jul 14 19:53:08.010747 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.010625 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-util\") pod \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " Jul 14 19:53:08.010747 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.010670 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp5s8\" (UniqueName: \"kubernetes.io/projected/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-kube-api-access-vp5s8\") pod \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\" (UID: \"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab\") " Jul 14 19:53:08.011110 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.011086 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-bundle" (OuterVolumeSpecName: "bundle") pod "5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" (UID: "5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:08.013290 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.013260 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-kube-api-access-vp5s8" (OuterVolumeSpecName: "kube-api-access-vp5s8") pod "5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" (UID: "5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab"). InnerVolumeSpecName "kube-api-access-vp5s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:08.015870 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.015849 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-util" (OuterVolumeSpecName: "util") pod "5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" (UID: "5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:08.111865 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.111827 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:08.111865 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.111859 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:08.111865 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.111871 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5s8\" (UniqueName: \"kubernetes.io/projected/5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab-kube-api-access-vp5s8\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:08.776618 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.776582 2553 generic.go:356] "Generic (PLEG): container finished" podID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerID="fac6f22d493e983d79117377dd138d1ea82a7d1188b91f9944083700f8a03aff" exitCode=0 Jul 14 19:53:08.777028 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.776662 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" event={"ID":"73c6013e-2f06-43bf-ab99-e3e14b27ea62","Type":"ContainerDied","Data":"fac6f22d493e983d79117377dd138d1ea82a7d1188b91f9944083700f8a03aff"} Jul 14 19:53:08.778476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.778458 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" Jul 14 19:53:08.778576 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.778490 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm" event={"ID":"5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab","Type":"ContainerDied","Data":"9a50cae421d109813e19f2a0e8416e1f456ea15d972aeaf5272616c94e6d5cb7"} Jul 14 19:53:08.778576 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.778530 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a50cae421d109813e19f2a0e8416e1f456ea15d972aeaf5272616c94e6d5cb7" Jul 14 19:53:08.780533 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.780508 2553 generic.go:356] "Generic (PLEG): container finished" podID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerID="965a62d6e0a8a2447592820191f3d18054a2c34d3f244d40c68cf3079e698eb9" exitCode=0 Jul 14 19:53:08.780651 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.780606 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" event={"ID":"a8b7f543-5c73-4adb-b711-13c87679d19c","Type":"ContainerDied","Data":"965a62d6e0a8a2447592820191f3d18054a2c34d3f244d40c68cf3079e698eb9"} Jul 14 19:53:08.909405 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:08.909382 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:09.016756 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.016723 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-bundle\") pod \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " Jul 14 19:53:09.016900 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.016778 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-util\") pod \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " Jul 14 19:53:09.016900 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.016808 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmpp8\" (UniqueName: \"kubernetes.io/projected/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-kube-api-access-nmpp8\") pod \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\" (UID: \"1e3e7d92-e7c4-4cc3-9435-1328fc7db435\") " Jul 14 19:53:09.017441 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.017414 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-bundle" (OuterVolumeSpecName: "bundle") pod "1e3e7d92-e7c4-4cc3-9435-1328fc7db435" (UID: "1e3e7d92-e7c4-4cc3-9435-1328fc7db435"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:09.018824 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.018800 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-kube-api-access-nmpp8" (OuterVolumeSpecName: "kube-api-access-nmpp8") pod "1e3e7d92-e7c4-4cc3-9435-1328fc7db435" (UID: "1e3e7d92-e7c4-4cc3-9435-1328fc7db435"). InnerVolumeSpecName "kube-api-access-nmpp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:09.022006 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.021984 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-util" (OuterVolumeSpecName: "util") pod "1e3e7d92-e7c4-4cc3-9435-1328fc7db435" (UID: "1e3e7d92-e7c4-4cc3-9435-1328fc7db435"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:09.117534 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.117503 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:09.117534 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.117530 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:09.117534 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.117539 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmpp8\" (UniqueName: \"kubernetes.io/projected/1e3e7d92-e7c4-4cc3-9435-1328fc7db435-kube-api-access-nmpp8\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:09.787394 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.787354 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" event={"ID":"1e3e7d92-e7c4-4cc3-9435-1328fc7db435","Type":"ContainerDied","Data":"350570a32b9ba7a45c66a75545fad4499e2b33b08b17e6f8a4497f4c48008fec"} Jul 14 19:53:09.787833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.787413 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="350570a32b9ba7a45c66a75545fad4499e2b33b08b17e6f8a4497f4c48008fec" Jul 14 19:53:09.787833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.787371 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk" Jul 14 19:53:09.948291 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.948268 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:09.951742 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:09.951716 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:10.012635 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.012602 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f95df6849-jmlpc"] Jul 14 19:53:10.025883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.025857 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvjs9\" (UniqueName: \"kubernetes.io/projected/73c6013e-2f06-43bf-ab99-e3e14b27ea62-kube-api-access-kvjs9\") pod \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " Jul 14 19:53:10.025883 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.025888 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-bundle\") pod \"a8b7f543-5c73-4adb-b711-13c87679d19c\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " Jul 14 19:53:10.026103 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.025917 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-util\") pod \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " Jul 14 19:53:10.026103 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.025981 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqkv\" (UniqueName: \"kubernetes.io/projected/a8b7f543-5c73-4adb-b711-13c87679d19c-kube-api-access-4nqkv\") pod \"a8b7f543-5c73-4adb-b711-13c87679d19c\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " Jul 14 19:53:10.026103 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.026055 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-util\") pod \"a8b7f543-5c73-4adb-b711-13c87679d19c\" (UID: \"a8b7f543-5c73-4adb-b711-13c87679d19c\") " Jul 14 19:53:10.026271 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.026136 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-bundle\") pod \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\" (UID: \"73c6013e-2f06-43bf-ab99-e3e14b27ea62\") " Jul 14 19:53:10.026779 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.026749 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-bundle" (OuterVolumeSpecName: "bundle") pod "73c6013e-2f06-43bf-ab99-e3e14b27ea62" (UID: "73c6013e-2f06-43bf-ab99-e3e14b27ea62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:10.026940 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.026910 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-bundle" (OuterVolumeSpecName: "bundle") pod "a8b7f543-5c73-4adb-b711-13c87679d19c" (UID: "a8b7f543-5c73-4adb-b711-13c87679d19c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:10.028537 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.028508 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b7f543-5c73-4adb-b711-13c87679d19c-kube-api-access-4nqkv" (OuterVolumeSpecName: "kube-api-access-4nqkv") pod "a8b7f543-5c73-4adb-b711-13c87679d19c" (UID: "a8b7f543-5c73-4adb-b711-13c87679d19c"). InnerVolumeSpecName "kube-api-access-4nqkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:10.028635 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.028581 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c6013e-2f06-43bf-ab99-e3e14b27ea62-kube-api-access-kvjs9" (OuterVolumeSpecName: "kube-api-access-kvjs9") pod "73c6013e-2f06-43bf-ab99-e3e14b27ea62" (UID: "73c6013e-2f06-43bf-ab99-e3e14b27ea62"). InnerVolumeSpecName "kube-api-access-kvjs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:10.032235 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.032208 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-util" (OuterVolumeSpecName: "util") pod "a8b7f543-5c73-4adb-b711-13c87679d19c" (UID: "a8b7f543-5c73-4adb-b711-13c87679d19c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:10.032901 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.032842 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-util" (OuterVolumeSpecName: "util") pod "73c6013e-2f06-43bf-ab99-e3e14b27ea62" (UID: "73c6013e-2f06-43bf-ab99-e3e14b27ea62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:10.126780 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.126755 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:10.126931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.126783 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:10.126931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.126794 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvjs9\" (UniqueName: \"kubernetes.io/projected/73c6013e-2f06-43bf-ab99-e3e14b27ea62-kube-api-access-kvjs9\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:10.126931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.126804 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8b7f543-5c73-4adb-b711-13c87679d19c-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:10.126931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.126811 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c6013e-2f06-43bf-ab99-e3e14b27ea62-util\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:10.126931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.126819 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqkv\" (UniqueName: \"kubernetes.io/projected/a8b7f543-5c73-4adb-b711-13c87679d19c-kube-api-access-4nqkv\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:10.135107 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.135077 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:10.234440 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.234405 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:10.294826 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.294795 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:10.794207 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.794178 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" Jul 14 19:53:10.794207 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.794184 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm" event={"ID":"a8b7f543-5c73-4adb-b711-13c87679d19c","Type":"ContainerDied","Data":"0e72bedf487fc8ed9ed891ae5e6a0c200723aafe81256c44c0d8dc66e6164750"} Jul 14 19:53:10.794660 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.794222 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e72bedf487fc8ed9ed891ae5e6a0c200723aafe81256c44c0d8dc66e6164750" Jul 14 19:53:10.796030 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.796006 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" event={"ID":"73c6013e-2f06-43bf-ab99-e3e14b27ea62","Type":"ContainerDied","Data":"507c6405a2008927d84685f85faa9b75df406d3ce1a99df2b7e7dd56a78e7d6e"} Jul 14 19:53:10.796156 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.796040 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507c6405a2008927d84685f85faa9b75df406d3ce1a99df2b7e7dd56a78e7d6e" Jul 14 19:53:10.796156 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:10.796041 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484" Jul 14 19:53:16.988217 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:16.988172 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:20.411667 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:20.411626 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:22.697568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:22.697537 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:24.222285 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:24.222242 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:24.416013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:24.415953 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:26.595126 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:26.595090 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:26.763622 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:26.763556 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:27.561110 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:27.561048 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:28.973024 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:28.972949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:31.426290 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:31.426214 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:31.551238 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:31.551206 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:32.097631 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:32.097571 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:32.170812 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:32.170780 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:35.033765 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.033706 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f95df6849-jmlpc" podUID="6ede2c99-11cb-4bf3-b82c-97831c28dbca" containerName="console" containerID="cri-o://9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702" gracePeriod=15 Jul 14 19:53:35.280089 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.280039 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:53:35.445027 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.444993 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-trusted-ca-bundle\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445244 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445038 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-oauth-config\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445244 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445089 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-service-ca\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445244 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445116 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-serving-cert\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445244 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445150 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-oauth-serving-cert\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445244 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445175 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp6xl\" (UniqueName: \"kubernetes.io/projected/6ede2c99-11cb-4bf3-b82c-97831c28dbca-kube-api-access-mp6xl\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445510 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445340 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-config\") pod \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\" (UID: \"6ede2c99-11cb-4bf3-b82c-97831c28dbca\") " Jul 14 19:53:35.445597 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445577 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:35.445597 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445583 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-service-ca" (OuterVolumeSpecName: "service-ca") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:35.445722 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445703 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:35.445813 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.445793 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-config" (OuterVolumeSpecName: "console-config") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:35.447442 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.447422 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:53:35.447831 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.447811 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:53:35.447900 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.447814 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ede2c99-11cb-4bf3-b82c-97831c28dbca-kube-api-access-mp6xl" (OuterVolumeSpecName: "kube-api-access-mp6xl") pod "6ede2c99-11cb-4bf3-b82c-97831c28dbca" (UID: "6ede2c99-11cb-4bf3-b82c-97831c28dbca"). InnerVolumeSpecName "kube-api-access-mp6xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:35.546851 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546810 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.546851 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546842 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-trusted-ca-bundle\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.546851 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546860 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-oauth-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.547136 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546871 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.547136 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546884 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ede2c99-11cb-4bf3-b82c-97831c28dbca-console-serving-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.547136 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546895 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ede2c99-11cb-4bf3-b82c-97831c28dbca-oauth-serving-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.547136 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.546905 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp6xl\" (UniqueName: \"kubernetes.io/projected/6ede2c99-11cb-4bf3-b82c-97831c28dbca-kube-api-access-mp6xl\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:53:35.919205 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.919169 2553 generic.go:356] "Generic (PLEG): container finished" podID="6ede2c99-11cb-4bf3-b82c-97831c28dbca" containerID="9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702" exitCode=0 Jul 14 19:53:35.919350 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.919219 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95df6849-jmlpc" Jul 14 19:53:35.919350 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.919264 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95df6849-jmlpc" event={"ID":"6ede2c99-11cb-4bf3-b82c-97831c28dbca","Type":"ContainerDied","Data":"9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702"} Jul 14 19:53:35.919350 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.919346 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95df6849-jmlpc" event={"ID":"6ede2c99-11cb-4bf3-b82c-97831c28dbca","Type":"ContainerDied","Data":"0ef22e385083c1b913f2efb41b25d564ed10018e6fedf04ba812f1b9d2cb9fa3"} Jul 14 19:53:35.919461 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.919370 2553 scope.go:117] "RemoveContainer" containerID="9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702" Jul 14 19:53:35.928774 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.928757 2553 scope.go:117] "RemoveContainer" containerID="9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702" Jul 14 19:53:35.929037 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:53:35.929002 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702\": container with ID starting with 9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702 not found: ID does not exist" containerID="9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702" Jul 14 19:53:35.929141 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.929045 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702"} err="failed to get container status \"9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702\": rpc error: code = NotFound desc = could not find container \"9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702\": container with ID starting with 9799a9de8068f65be20ca1dd9de33a35343b8424c85f0c34927f54ad9d7f1702 not found: ID does not exist" Jul 14 19:53:35.941174 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.941144 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f95df6849-jmlpc"] Jul 14 19:53:35.946910 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:35.946881 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f95df6849-jmlpc"] Jul 14 19:53:37.035975 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:37.035942 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ede2c99-11cb-4bf3-b82c-97831c28dbca" path="/var/lib/kubelet/pods/6ede2c99-11cb-4bf3-b82c-97831c28dbca/volumes" Jul 14 19:53:40.382918 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:40.382883 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:40.499850 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:40.499821 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:44.561837 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:44.561806 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:45.357248 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:45.357207 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:49.361623 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:49.361590 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:49.432370 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:49.432339 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:49.969345 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:49.969313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:50.070781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:50.070752 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:55.350798 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:55.350766 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:55.409479 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:55.409449 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:59.704905 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:59.704834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:59.806108 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:53:59.806055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:01.636529 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:01.636492 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:01.701321 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:01.701290 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:06.407634 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:06.407593 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:06.508895 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:06.508865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:11.202278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:11.202246 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:11.298408 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:11.298381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:23.937893 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:23.937857 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:23.997453 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:23.997418 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:26.921348 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.921306 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:26.933827 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.933797 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:26.934124 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934112 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="extract" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934125 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="extract" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934135 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="util" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934141 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="util" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934146 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="extract" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934151 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="extract" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934157 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ede2c99-11cb-4bf3-b82c-97831c28dbca" containerName="console" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934162 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ede2c99-11cb-4bf3-b82c-97831c28dbca" containerName="console" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934168 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="pull" Jul 14 19:54:26.934171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934173 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934179 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934183 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934189 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934193 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934200 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934209 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934219 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="util" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934224 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="util" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934229 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934234 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934239 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="util" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934251 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="util" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934258 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="util" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934262 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="util" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934267 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934273 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="pull" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934329 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934338 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3e7d92-e7c4-4cc3-9435-1328fc7db435" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934344 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b7f543-5c73-4adb-b711-13c87679d19c" containerName="extract" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934349 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ede2c99-11cb-4bf3-b82c-97831c28dbca" containerName="console" Jul 14 19:54:26.934422 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.934354 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c6013e-2f06-43bf-ab99-e3e14b27ea62" containerName="extract" Jul 14 19:54:26.937381 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.937361 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:26.939973 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.939945 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slfsg\"" Jul 14 19:54:26.939973 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.939963 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 19:54:26.940171 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.940011 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:54:26.940771 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.940757 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:54:26.951050 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:26.951023 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:27.019931 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.019897 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:27.031270 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.031237 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:27.067109 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.067049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8hm\" (UniqueName: \"kubernetes.io/projected/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-kube-api-access-pd8hm\") pod \"limitador-limitador-64c94fd64f-j7q69\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.067282 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.067188 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-config-file\") pod \"limitador-limitador-64c94fd64f-j7q69\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.131509 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.131474 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:27.168417 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.168387 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8hm\" (UniqueName: \"kubernetes.io/projected/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-kube-api-access-pd8hm\") pod \"limitador-limitador-64c94fd64f-j7q69\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.168624 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.168451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-config-file\") pod \"limitador-limitador-64c94fd64f-j7q69\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.169095 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.169051 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-config-file\") pod \"limitador-limitador-64c94fd64f-j7q69\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.177645 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.177591 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8hm\" (UniqueName: \"kubernetes.io/projected/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-kube-api-access-pd8hm\") pod \"limitador-limitador-64c94fd64f-j7q69\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.247423 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.247384 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:27.382502 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.382477 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:27.383846 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:54:27.383814 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7be4b61_eb5a_42f1_b87b_7d2d2fbfd882.slice/crio-5c579e125a78bac0a8d2382b3e6d0227e9b7bdf61ee6cf7d133e17889bb4085f WatchSource:0}: Error finding container 5c579e125a78bac0a8d2382b3e6d0227e9b7bdf61ee6cf7d133e17889bb4085f: Status 404 returned error can't find the container with id 5c579e125a78bac0a8d2382b3e6d0227e9b7bdf61ee6cf7d133e17889bb4085f Jul 14 19:54:27.748057 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.748022 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qhlxl"] Jul 14 19:54:27.751247 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.751227 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:27.753964 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.753939 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5q7rj\"" Jul 14 19:54:27.756225 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.756202 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhlxl"] Jul 14 19:54:27.786313 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.786280 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:27.873963 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.873922 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59nc\" (UniqueName: \"kubernetes.io/projected/2e1473cb-3cee-420e-848a-559625521fa1-kube-api-access-w59nc\") pod \"authorino-58cd955675-qhlxl\" (UID: \"2e1473cb-3cee-420e-848a-559625521fa1\") " pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:27.887613 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.887586 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:27.974643 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.974605 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59nc\" (UniqueName: \"kubernetes.io/projected/2e1473cb-3cee-420e-848a-559625521fa1-kube-api-access-w59nc\") pod \"authorino-58cd955675-qhlxl\" (UID: \"2e1473cb-3cee-420e-848a-559625521fa1\") " pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:27.986196 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:27.986165 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59nc\" (UniqueName: \"kubernetes.io/projected/2e1473cb-3cee-420e-848a-559625521fa1-kube-api-access-w59nc\") pod \"authorino-58cd955675-qhlxl\" (UID: \"2e1473cb-3cee-420e-848a-559625521fa1\") " pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:28.061492 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:28.061415 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:28.185514 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:28.185471 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" event={"ID":"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882","Type":"ContainerStarted","Data":"5c579e125a78bac0a8d2382b3e6d0227e9b7bdf61ee6cf7d133e17889bb4085f"} Jul 14 19:54:28.206703 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:28.206671 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhlxl"] Jul 14 19:54:28.209261 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:54:28.209231 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e1473cb_3cee_420e_848a_559625521fa1.slice/crio-7bee936d86755c8747c3fbb1f027be1898ec09ddd55f49da5cfce609f7f5edd8 WatchSource:0}: Error finding container 7bee936d86755c8747c3fbb1f027be1898ec09ddd55f49da5cfce609f7f5edd8: Status 404 returned error can't find the container with id 7bee936d86755c8747c3fbb1f027be1898ec09ddd55f49da5cfce609f7f5edd8 Jul 14 19:54:29.195163 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:29.195122 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhlxl" event={"ID":"2e1473cb-3cee-420e-848a-559625521fa1","Type":"ContainerStarted","Data":"7bee936d86755c8747c3fbb1f027be1898ec09ddd55f49da5cfce609f7f5edd8"} Jul 14 19:54:32.211712 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:32.211610 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" event={"ID":"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882","Type":"ContainerStarted","Data":"cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1"} Jul 14 19:54:32.212158 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:32.211774 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:32.213027 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:32.213006 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhlxl" event={"ID":"2e1473cb-3cee-420e-848a-559625521fa1","Type":"ContainerStarted","Data":"1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34"} Jul 14 19:54:32.230510 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:32.230465 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" podStartSLOduration=1.722170542 podStartE2EDuration="6.230447913s" totalImagesPullingTime="4.508277371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:32.229321085 +0000 UTC m=+477.677947530" watchObservedRunningTime="2026-07-14 19:54:32.230447913 +0000 UTC m=+477.679074357" Jul 14 19:54:32.247866 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:32.247813 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qhlxl" podStartSLOduration=1.564477719 podStartE2EDuration="5.247798034s" totalImagesPullingTime="3.683320315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:32.246089576 +0000 UTC m=+477.694716017" watchObservedRunningTime="2026-07-14 19:54:32.247798034 +0000 UTC m=+477.696424478" Jul 14 19:54:32.374383 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:32.374348 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhlxl"] Jul 14 19:54:34.223991 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:34.223953 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qhlxl" podUID="2e1473cb-3cee-420e-848a-559625521fa1" containerName="authorino" containerID="cri-o://1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34" gracePeriod=30 Jul 14 19:54:34.464628 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:34.464604 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:34.637018 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:34.636983 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w59nc\" (UniqueName: \"kubernetes.io/projected/2e1473cb-3cee-420e-848a-559625521fa1-kube-api-access-w59nc\") pod \"2e1473cb-3cee-420e-848a-559625521fa1\" (UID: \"2e1473cb-3cee-420e-848a-559625521fa1\") " Jul 14 19:54:34.639107 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:34.639077 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1473cb-3cee-420e-848a-559625521fa1-kube-api-access-w59nc" (OuterVolumeSpecName: "kube-api-access-w59nc") pod "2e1473cb-3cee-420e-848a-559625521fa1" (UID: "2e1473cb-3cee-420e-848a-559625521fa1"). InnerVolumeSpecName "kube-api-access-w59nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:34.737633 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:34.737583 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59nc\" (UniqueName: \"kubernetes.io/projected/2e1473cb-3cee-420e-848a-559625521fa1-kube-api-access-w59nc\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:54:35.233845 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.233758 2553 generic.go:356] "Generic (PLEG): container finished" podID="2e1473cb-3cee-420e-848a-559625521fa1" containerID="1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34" exitCode=0 Jul 14 19:54:35.233845 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.233802 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhlxl" Jul 14 19:54:35.233845 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.233836 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhlxl" event={"ID":"2e1473cb-3cee-420e-848a-559625521fa1","Type":"ContainerDied","Data":"1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34"} Jul 14 19:54:35.234380 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.233878 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhlxl" event={"ID":"2e1473cb-3cee-420e-848a-559625521fa1","Type":"ContainerDied","Data":"7bee936d86755c8747c3fbb1f027be1898ec09ddd55f49da5cfce609f7f5edd8"} Jul 14 19:54:35.234380 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.233895 2553 scope.go:117] "RemoveContainer" containerID="1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34" Jul 14 19:54:35.241520 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.241499 2553 scope.go:117] "RemoveContainer" containerID="1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34" Jul 14 19:54:35.241736 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:54:35.241718 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34\": container with ID starting with 1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34 not found: ID does not exist" containerID="1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34" Jul 14 19:54:35.241799 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.241749 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34"} err="failed to get container status \"1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34\": rpc error: code = NotFound desc = could not find container \"1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34\": container with ID starting with 1fbb1f11d433eb2a30a0e9f6ec85520fe2f34cead90ee470b0614c836367bb34 not found: ID does not exist" Jul 14 19:54:35.252667 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.252627 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhlxl"] Jul 14 19:54:35.254801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:35.254780 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhlxl"] Jul 14 19:54:37.036148 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:37.036117 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1473cb-3cee-420e-848a-559625521fa1" path="/var/lib/kubelet/pods/2e1473cb-3cee-420e-848a-559625521fa1/volumes" Jul 14 19:54:43.220797 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:43.220767 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:43.372627 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:43.372596 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:43.372855 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:43.372801 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" podUID="d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" containerName="limitador" containerID="cri-o://cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1" gracePeriod=30 Jul 14 19:54:43.917549 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:43.917521 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:43.920277 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:43.920255 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-config-file\") pod \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " Jul 14 19:54:43.920563 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:43.920544 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-config-file" (OuterVolumeSpecName: "config-file") pod "d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" (UID: "d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:54:44.021388 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.021308 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd8hm\" (UniqueName: \"kubernetes.io/projected/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-kube-api-access-pd8hm\") pod \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\" (UID: \"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882\") " Jul 14 19:54:44.021546 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.021496 2553 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-config-file\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:54:44.023281 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.023257 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-kube-api-access-pd8hm" (OuterVolumeSpecName: "kube-api-access-pd8hm") pod "d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" (UID: "d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882"). InnerVolumeSpecName "kube-api-access-pd8hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:44.122098 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.122047 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd8hm\" (UniqueName: \"kubernetes.io/projected/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882-kube-api-access-pd8hm\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:54:44.283862 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.283780 2553 generic.go:356] "Generic (PLEG): container finished" podID="d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" containerID="cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1" exitCode=0 Jul 14 19:54:44.283862 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.283828 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" Jul 14 19:54:44.283862 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.283850 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" event={"ID":"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882","Type":"ContainerDied","Data":"cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1"} Jul 14 19:54:44.284394 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.283892 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j7q69" event={"ID":"d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882","Type":"ContainerDied","Data":"5c579e125a78bac0a8d2382b3e6d0227e9b7bdf61ee6cf7d133e17889bb4085f"} Jul 14 19:54:44.284394 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.283913 2553 scope.go:117] "RemoveContainer" containerID="cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1" Jul 14 19:54:44.292928 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.292910 2553 scope.go:117] "RemoveContainer" containerID="cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1" Jul 14 19:54:44.293231 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:54:44.293212 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1\": container with ID starting with cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1 not found: ID does not exist" containerID="cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1" Jul 14 19:54:44.293287 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.293238 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1"} err="failed to get container status \"cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1\": rpc error: code = NotFound desc = could not find container \"cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1\": container with ID starting with cc0d4945ca5bb5b59144dd4c7ab3f3e1a51f47c8f6d23b9791c86cea20533cf1 not found: ID does not exist" Jul 14 19:54:44.306162 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.306046 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:44.312325 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:44.312300 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j7q69"] Jul 14 19:54:45.035956 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:45.035926 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" path="/var/lib/kubelet/pods/d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882/volumes" Jul 14 19:54:48.124654 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.124609 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:48.246188 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.246145 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:48.585912 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.585877 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-676fd"] Jul 14 19:54:48.586211 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.586199 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" containerName="limitador" Jul 14 19:54:48.586259 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.586212 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" containerName="limitador" Jul 14 19:54:48.586259 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.586224 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1473cb-3cee-420e-848a-559625521fa1" containerName="authorino" Jul 14 19:54:48.586259 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.586230 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1473cb-3cee-420e-848a-559625521fa1" containerName="authorino" Jul 14 19:54:48.586344 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.586288 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7be4b61-eb5a-42f1-b87b-7d2d2fbfd882" containerName="limitador" Jul 14 19:54:48.586344 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.586298 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1473cb-3cee-420e-848a-559625521fa1" containerName="authorino" Jul 14 19:54:48.592406 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.592383 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.596338 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.596320 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 19:54:48.597418 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.597324 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 19:54:48.597418 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.597348 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 19:54:48.597672 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.597650 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kk7j8\"" Jul 14 19:54:48.599981 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.599961 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-676fd"] Jul 14 19:54:48.618795 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.618768 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:48.659028 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.658996 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wssqp\" (UniqueName: \"kubernetes.io/projected/87a76e58-b687-4c45-89ae-0a50dc6bc486-kube-api-access-wssqp\") pod \"postgres-69fb4594cf-676fd\" (UID: \"87a76e58-b687-4c45-89ae-0a50dc6bc486\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.659194 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.659087 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/87a76e58-b687-4c45-89ae-0a50dc6bc486-data\") pod \"postgres-69fb4594cf-676fd\" (UID: \"87a76e58-b687-4c45-89ae-0a50dc6bc486\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.759879 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.759848 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/87a76e58-b687-4c45-89ae-0a50dc6bc486-data\") pod \"postgres-69fb4594cf-676fd\" (UID: \"87a76e58-b687-4c45-89ae-0a50dc6bc486\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.760080 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.759900 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wssqp\" (UniqueName: \"kubernetes.io/projected/87a76e58-b687-4c45-89ae-0a50dc6bc486-kube-api-access-wssqp\") pod \"postgres-69fb4594cf-676fd\" (UID: \"87a76e58-b687-4c45-89ae-0a50dc6bc486\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.760274 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.760256 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/87a76e58-b687-4c45-89ae-0a50dc6bc486-data\") pod \"postgres-69fb4594cf-676fd\" (UID: \"87a76e58-b687-4c45-89ae-0a50dc6bc486\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.769383 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.769353 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wssqp\" (UniqueName: \"kubernetes.io/projected/87a76e58-b687-4c45-89ae-0a50dc6bc486-kube-api-access-wssqp\") pod \"postgres-69fb4594cf-676fd\" (UID: \"87a76e58-b687-4c45-89ae-0a50dc6bc486\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:48.903140 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:48.903100 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:49.036602 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:54:49.036566 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a76e58_b687_4c45_89ae_0a50dc6bc486.slice/crio-bcaddabb5124dadf78e6776ca70bec26969aadc8f94ebeff77d0505f0fca4f4b WatchSource:0}: Error finding container bcaddabb5124dadf78e6776ca70bec26969aadc8f94ebeff77d0505f0fca4f4b: Status 404 returned error can't find the container with id bcaddabb5124dadf78e6776ca70bec26969aadc8f94ebeff77d0505f0fca4f4b Jul 14 19:54:49.036822 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:49.036804 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-676fd"] Jul 14 19:54:49.309973 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:49.309891 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" event={"ID":"87a76e58-b687-4c45-89ae-0a50dc6bc486","Type":"ContainerStarted","Data":"bcaddabb5124dadf78e6776ca70bec26969aadc8f94ebeff77d0505f0fca4f4b"} Jul 14 19:54:55.346817 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:55.346783 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" event={"ID":"87a76e58-b687-4c45-89ae-0a50dc6bc486","Type":"ContainerStarted","Data":"8b5da83bfec0de70620377fb1b3f92786da24a1c2f9041a80cdaaff9eb0c8aeb"} Jul 14 19:54:55.347314 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:55.346844 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:54:55.364650 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:54:55.364606 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" podStartSLOduration=1.994623366 podStartE2EDuration="7.364581448s" totalImagesPullingTime="5.369958082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:55.363793229 +0000 UTC m=+500.812419697" watchObservedRunningTime="2026-07-14 19:54:55.364581448 +0000 UTC m=+500.813207893" Jul 14 19:55:00.768994 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:00.768956 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:00.828821 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:00.828786 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:01.380910 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.380875 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-676fd" Jul 14 19:55:01.692735 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.692611 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:01.910176 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.910147 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:01.922351 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.922317 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qdwn9"] Jul 14 19:55:01.925533 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.925512 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:01.928096 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.928077 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:55:01.928188 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.928109 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5q7rj\"" Jul 14 19:55:01.928188 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.928109 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:55:01.932868 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.932848 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qdwn9"] Jul 14 19:55:01.982251 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:01.982162 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ksm\" (UniqueName: \"kubernetes.io/projected/34d5fda3-0005-43a9-850d-97408681fa24-kube-api-access-65ksm\") pod \"authorino-5d4b67c446-qdwn9\" (UID: \"34d5fda3-0005-43a9-850d-97408681fa24\") " pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:02.083397 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.083358 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ksm\" (UniqueName: \"kubernetes.io/projected/34d5fda3-0005-43a9-850d-97408681fa24-kube-api-access-65ksm\") pod \"authorino-5d4b67c446-qdwn9\" (UID: \"34d5fda3-0005-43a9-850d-97408681fa24\") " pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:02.092419 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.092387 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ksm\" (UniqueName: \"kubernetes.io/projected/34d5fda3-0005-43a9-850d-97408681fa24-kube-api-access-65ksm\") pod \"authorino-5d4b67c446-qdwn9\" (UID: \"34d5fda3-0005-43a9-850d-97408681fa24\") " pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:02.168534 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.168505 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qdwn9"] Jul 14 19:55:02.168770 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.168756 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:02.295224 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.295199 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qdwn9"] Jul 14 19:55:02.297295 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:55:02.297272 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34d5fda3_0005_43a9_850d_97408681fa24.slice/crio-6f7047dcf6589172534abdb0a8d40d0a0dbc8732654c7dcdc41237e3860ba732 WatchSource:0}: Error finding container 6f7047dcf6589172534abdb0a8d40d0a0dbc8732654c7dcdc41237e3860ba732: Status 404 returned error can't find the container with id 6f7047dcf6589172534abdb0a8d40d0a0dbc8732654c7dcdc41237e3860ba732 Jul 14 19:55:02.347945 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.347909 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-869cf89489-m7dx2"] Jul 14 19:55:02.352293 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.352273 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.354664 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.354646 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 19:55:02.359040 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.359015 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-869cf89489-m7dx2"] Jul 14 19:55:02.381960 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.381925 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" event={"ID":"34d5fda3-0005-43a9-850d-97408681fa24","Type":"ContainerStarted","Data":"6f7047dcf6589172534abdb0a8d40d0a0dbc8732654c7dcdc41237e3860ba732"} Jul 14 19:55:02.487547 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.487516 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29b6s\" (UniqueName: \"kubernetes.io/projected/621eefc2-1860-43de-bb4f-9a7e748215ae-kube-api-access-29b6s\") pod \"authorino-869cf89489-m7dx2\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.487741 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.487590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/621eefc2-1860-43de-bb4f-9a7e748215ae-tls-cert\") pod \"authorino-869cf89489-m7dx2\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.588794 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.588709 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/621eefc2-1860-43de-bb4f-9a7e748215ae-tls-cert\") pod \"authorino-869cf89489-m7dx2\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.588794 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.588763 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29b6s\" (UniqueName: \"kubernetes.io/projected/621eefc2-1860-43de-bb4f-9a7e748215ae-kube-api-access-29b6s\") pod \"authorino-869cf89489-m7dx2\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.591027 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.591004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/621eefc2-1860-43de-bb4f-9a7e748215ae-tls-cert\") pod \"authorino-869cf89489-m7dx2\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.597718 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.597690 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29b6s\" (UniqueName: \"kubernetes.io/projected/621eefc2-1860-43de-bb4f-9a7e748215ae-kube-api-access-29b6s\") pod \"authorino-869cf89489-m7dx2\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.662592 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.662555 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:55:02.787444 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:02.787416 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-869cf89489-m7dx2"] Jul 14 19:55:02.789702 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:55:02.789678 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod621eefc2_1860_43de_bb4f_9a7e748215ae.slice/crio-68d8850cc239d319fda7b4a8071ee7183ccb50f249def639fbbdb76c4f82c45a WatchSource:0}: Error finding container 68d8850cc239d319fda7b4a8071ee7183ccb50f249def639fbbdb76c4f82c45a: Status 404 returned error can't find the container with id 68d8850cc239d319fda7b4a8071ee7183ccb50f249def639fbbdb76c4f82c45a Jul 14 19:55:03.389737 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.389685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-869cf89489-m7dx2" event={"ID":"621eefc2-1860-43de-bb4f-9a7e748215ae","Type":"ContainerStarted","Data":"da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc"} Jul 14 19:55:03.389737 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.389734 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-869cf89489-m7dx2" event={"ID":"621eefc2-1860-43de-bb4f-9a7e748215ae","Type":"ContainerStarted","Data":"68d8850cc239d319fda7b4a8071ee7183ccb50f249def639fbbdb76c4f82c45a"} Jul 14 19:55:03.391466 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.391437 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" event={"ID":"34d5fda3-0005-43a9-850d-97408681fa24","Type":"ContainerStarted","Data":"3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9"} Jul 14 19:55:03.391570 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.391502 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" podUID="34d5fda3-0005-43a9-850d-97408681fa24" containerName="authorino" containerID="cri-o://3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9" gracePeriod=30 Jul 14 19:55:03.409302 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.409243 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-869cf89489-m7dx2" podStartSLOduration=0.903409053 podStartE2EDuration="1.409222113s" totalImagesPullingTime="505.81306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:03.4052399 +0000 UTC m=+508.853866379" watchObservedRunningTime="2026-07-14 19:55:03.409222113 +0000 UTC m=+508.857848560" Jul 14 19:55:03.423194 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.423141 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" podStartSLOduration=1.86445065 podStartE2EDuration="2.423124977s" totalImagesPullingTime="558.674327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:03.420155012 +0000 UTC m=+508.868781458" watchObservedRunningTime="2026-07-14 19:55:03.423124977 +0000 UTC m=+508.871751423" Jul 14 19:55:03.621164 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.621139 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:03.697129 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.697090 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65ksm\" (UniqueName: \"kubernetes.io/projected/34d5fda3-0005-43a9-850d-97408681fa24-kube-api-access-65ksm\") pod \"34d5fda3-0005-43a9-850d-97408681fa24\" (UID: \"34d5fda3-0005-43a9-850d-97408681fa24\") " Jul 14 19:55:03.699054 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.699030 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d5fda3-0005-43a9-850d-97408681fa24-kube-api-access-65ksm" (OuterVolumeSpecName: "kube-api-access-65ksm") pod "34d5fda3-0005-43a9-850d-97408681fa24" (UID: "34d5fda3-0005-43a9-850d-97408681fa24"). InnerVolumeSpecName "kube-api-access-65ksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:55:03.798695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:03.798642 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65ksm\" (UniqueName: \"kubernetes.io/projected/34d5fda3-0005-43a9-850d-97408681fa24-kube-api-access-65ksm\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:55:04.397013 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.396980 2553 generic.go:356] "Generic (PLEG): container finished" podID="34d5fda3-0005-43a9-850d-97408681fa24" containerID="3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9" exitCode=0 Jul 14 19:55:04.397440 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.397023 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" Jul 14 19:55:04.397440 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.397082 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" event={"ID":"34d5fda3-0005-43a9-850d-97408681fa24","Type":"ContainerDied","Data":"3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9"} Jul 14 19:55:04.397440 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.397120 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qdwn9" event={"ID":"34d5fda3-0005-43a9-850d-97408681fa24","Type":"ContainerDied","Data":"6f7047dcf6589172534abdb0a8d40d0a0dbc8732654c7dcdc41237e3860ba732"} Jul 14 19:55:04.397440 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.397136 2553 scope.go:117] "RemoveContainer" containerID="3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9" Jul 14 19:55:04.405511 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.405495 2553 scope.go:117] "RemoveContainer" containerID="3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9" Jul 14 19:55:04.405766 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:55:04.405749 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9\": container with ID starting with 3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9 not found: ID does not exist" containerID="3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9" Jul 14 19:55:04.405824 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.405774 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9"} err="failed to get container status \"3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9\": rpc error: code = NotFound desc = could not find container \"3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9\": container with ID starting with 3d0cd5505bbe500db25e087b705ce0ccf67e74f58cf598b66f5c5cd94dc4eab9 not found: ID does not exist" Jul 14 19:55:04.417937 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.417912 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qdwn9"] Jul 14 19:55:04.422372 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:04.422350 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qdwn9"] Jul 14 19:55:05.037501 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:05.037468 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d5fda3-0005-43a9-850d-97408681fa24" path="/var/lib/kubelet/pods/34d5fda3-0005-43a9-850d-97408681fa24/volumes" Jul 14 19:55:08.033368 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.033334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:08.059598 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.059567 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9c85946f4-qtcv4"] Jul 14 19:55:08.059890 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.059878 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d5fda3-0005-43a9-850d-97408681fa24" containerName="authorino" Jul 14 19:55:08.059942 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.059892 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d5fda3-0005-43a9-850d-97408681fa24" containerName="authorino" Jul 14 19:55:08.059979 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.059958 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d5fda3-0005-43a9-850d-97408681fa24" containerName="authorino" Jul 14 19:55:08.064407 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.064390 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.066695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.066675 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 19:55:08.067165 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.067138 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 19:55:08.067330 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.067304 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-llx58\"" Jul 14 19:55:08.076145 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.076121 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c85946f4-qtcv4"] Jul 14 19:55:08.116414 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.116381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:08.138602 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.138557 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e14e89e7-fb2e-44fa-9927-b4207e744223-webhook-cert\") pod \"maas-controller-9c85946f4-qtcv4\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.138706 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.138670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbgz\" (UniqueName: \"kubernetes.io/projected/e14e89e7-fb2e-44fa-9927-b4207e744223-kube-api-access-blbgz\") pod \"maas-controller-9c85946f4-qtcv4\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.214561 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.214526 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fc5448547-s87x6"] Jul 14 19:55:08.217833 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.217815 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.228296 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.228272 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fc5448547-s87x6"] Jul 14 19:55:08.240010 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.239985 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e14e89e7-fb2e-44fa-9927-b4207e744223-webhook-cert\") pod \"maas-controller-9c85946f4-qtcv4\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.240176 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.240072 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbgz\" (UniqueName: \"kubernetes.io/projected/e14e89e7-fb2e-44fa-9927-b4207e744223-kube-api-access-blbgz\") pod \"maas-controller-9c85946f4-qtcv4\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.242303 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.242279 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e14e89e7-fb2e-44fa-9927-b4207e744223-webhook-cert\") pod \"maas-controller-9c85946f4-qtcv4\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.249076 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.249030 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbgz\" (UniqueName: \"kubernetes.io/projected/e14e89e7-fb2e-44fa-9927-b4207e744223-kube-api-access-blbgz\") pod \"maas-controller-9c85946f4-qtcv4\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.276346 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.276317 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:08.336104 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.335986 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:08.341083 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.341041 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772c7b9-298f-4c86-b564-1690a321ce87-webhook-cert\") pod \"maas-controller-fc5448547-s87x6\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.341206 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.341162 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4dh\" (UniqueName: \"kubernetes.io/projected/5772c7b9-298f-4c86-b564-1690a321ce87-kube-api-access-bk4dh\") pod \"maas-controller-fc5448547-s87x6\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.379769 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.379737 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:08.441572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.441542 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4dh\" (UniqueName: \"kubernetes.io/projected/5772c7b9-298f-4c86-b564-1690a321ce87-kube-api-access-bk4dh\") pod \"maas-controller-fc5448547-s87x6\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.441750 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.441602 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772c7b9-298f-4c86-b564-1690a321ce87-webhook-cert\") pod \"maas-controller-fc5448547-s87x6\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.445111 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.445082 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772c7b9-298f-4c86-b564-1690a321ce87-webhook-cert\") pod \"maas-controller-fc5448547-s87x6\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.451981 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.451922 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4dh\" (UniqueName: \"kubernetes.io/projected/5772c7b9-298f-4c86-b564-1690a321ce87-kube-api-access-bk4dh\") pod \"maas-controller-fc5448547-s87x6\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.511720 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.511665 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c85946f4-qtcv4"] Jul 14 19:55:08.514032 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:55:08.514008 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14e89e7_fb2e_44fa_9927_b4207e744223.slice/crio-d1f3681c13ef221051c5348e34078ebc6ce016c4818cb2633c914a8332a038c7 WatchSource:0}: Error finding container d1f3681c13ef221051c5348e34078ebc6ce016c4818cb2633c914a8332a038c7: Status 404 returned error can't find the container with id d1f3681c13ef221051c5348e34078ebc6ce016c4818cb2633c914a8332a038c7 Jul 14 19:55:08.528839 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.528818 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:08.687725 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:08.687696 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fc5448547-s87x6"] Jul 14 19:55:08.689883 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:55:08.689854 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5772c7b9_298f_4c86_b564_1690a321ce87.slice/crio-fc0b26155082fcd48e67436cc1f8e97f72c569e33e6d985d87e88faef36f595b WatchSource:0}: Error finding container fc0b26155082fcd48e67436cc1f8e97f72c569e33e6d985d87e88faef36f595b: Status 404 returned error can't find the container with id fc0b26155082fcd48e67436cc1f8e97f72c569e33e6d985d87e88faef36f595b Jul 14 19:55:09.428211 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:09.428110 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fc5448547-s87x6" event={"ID":"5772c7b9-298f-4c86-b564-1690a321ce87","Type":"ContainerStarted","Data":"fc0b26155082fcd48e67436cc1f8e97f72c569e33e6d985d87e88faef36f595b"} Jul 14 19:55:09.430342 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:09.430272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c85946f4-qtcv4" event={"ID":"e14e89e7-fb2e-44fa-9927-b4207e744223","Type":"ContainerStarted","Data":"d1f3681c13ef221051c5348e34078ebc6ce016c4818cb2633c914a8332a038c7"} Jul 14 19:55:10.372707 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:10.372671 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:10.428418 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:10.428380 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:10.847152 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:10.847121 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:10.949311 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:10.949278 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:11.442379 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:11.442345 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fc5448547-s87x6" event={"ID":"5772c7b9-298f-4c86-b564-1690a321ce87","Type":"ContainerStarted","Data":"2024ff0d92ffee84897abb7be4295af5f4a653a422401ffea77427fba8f1d341"} Jul 14 19:55:11.442807 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:11.442525 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:11.443770 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:11.443747 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c85946f4-qtcv4" event={"ID":"e14e89e7-fb2e-44fa-9927-b4207e744223","Type":"ContainerStarted","Data":"789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264"} Jul 14 19:55:11.443851 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:11.443790 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:11.472294 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:11.472246 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fc5448547-s87x6" podStartSLOduration=1.529368326 podStartE2EDuration="3.472229529s" totalImagesPullingTime="1.942861203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:11.46848015 +0000 UTC m=+516.917106596" watchObservedRunningTime="2026-07-14 19:55:11.472229529 +0000 UTC m=+516.920855998" Jul 14 19:55:11.497664 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:11.497617 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9c85946f4-qtcv4" podStartSLOduration=1.38955407 podStartE2EDuration="3.497602504s" totalImagesPullingTime="2.108048434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:11.496233149 +0000 UTC m=+516.944859622" watchObservedRunningTime="2026-07-14 19:55:11.497602504 +0000 UTC m=+516.946228972" Jul 14 19:55:22.451835 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.451805 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:22.452330 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.452103 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:55:22.518624 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.518585 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c85946f4-qtcv4"] Jul 14 19:55:22.518834 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.518806 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9c85946f4-qtcv4" podUID="e14e89e7-fb2e-44fa-9927-b4207e744223" containerName="manager" containerID="cri-o://789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264" gracePeriod=30 Jul 14 19:55:22.758485 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.758458 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:22.874951 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.874913 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e14e89e7-fb2e-44fa-9927-b4207e744223-webhook-cert\") pod \"e14e89e7-fb2e-44fa-9927-b4207e744223\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " Jul 14 19:55:22.875185 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.874977 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blbgz\" (UniqueName: \"kubernetes.io/projected/e14e89e7-fb2e-44fa-9927-b4207e744223-kube-api-access-blbgz\") pod \"e14e89e7-fb2e-44fa-9927-b4207e744223\" (UID: \"e14e89e7-fb2e-44fa-9927-b4207e744223\") " Jul 14 19:55:22.877148 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.877113 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14e89e7-fb2e-44fa-9927-b4207e744223-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e14e89e7-fb2e-44fa-9927-b4207e744223" (UID: "e14e89e7-fb2e-44fa-9927-b4207e744223"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:55:22.877260 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.877141 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14e89e7-fb2e-44fa-9927-b4207e744223-kube-api-access-blbgz" (OuterVolumeSpecName: "kube-api-access-blbgz") pod "e14e89e7-fb2e-44fa-9927-b4207e744223" (UID: "e14e89e7-fb2e-44fa-9927-b4207e744223"). InnerVolumeSpecName "kube-api-access-blbgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:55:22.975873 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.975831 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e14e89e7-fb2e-44fa-9927-b4207e744223-webhook-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:55:22.975873 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:22.975864 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blbgz\" (UniqueName: \"kubernetes.io/projected/e14e89e7-fb2e-44fa-9927-b4207e744223-kube-api-access-blbgz\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:55:23.503878 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.503841 2553 generic.go:356] "Generic (PLEG): container finished" podID="e14e89e7-fb2e-44fa-9927-b4207e744223" containerID="789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264" exitCode=0 Jul 14 19:55:23.504312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.503887 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c85946f4-qtcv4" Jul 14 19:55:23.504312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.503898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c85946f4-qtcv4" event={"ID":"e14e89e7-fb2e-44fa-9927-b4207e744223","Type":"ContainerDied","Data":"789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264"} Jul 14 19:55:23.504312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.503939 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c85946f4-qtcv4" event={"ID":"e14e89e7-fb2e-44fa-9927-b4207e744223","Type":"ContainerDied","Data":"d1f3681c13ef221051c5348e34078ebc6ce016c4818cb2633c914a8332a038c7"} Jul 14 19:55:23.504312 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.503961 2553 scope.go:117] "RemoveContainer" containerID="789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264" Jul 14 19:55:23.512090 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.512072 2553 scope.go:117] "RemoveContainer" containerID="789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264" Jul 14 19:55:23.512320 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:55:23.512297 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264\": container with ID starting with 789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264 not found: ID does not exist" containerID="789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264" Jul 14 19:55:23.512415 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.512325 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264"} err="failed to get container status \"789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264\": rpc error: code = NotFound desc = could not find container \"789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264\": container with ID starting with 789420f335dae8b4bfe48ecdd868d3b64e6e94519eac99ad27042ef3caa7c264 not found: ID does not exist" Jul 14 19:55:23.523517 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.523490 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c85946f4-qtcv4"] Jul 14 19:55:23.528131 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:23.528105 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9c85946f4-qtcv4"] Jul 14 19:55:25.036798 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:25.036762 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14e89e7-fb2e-44fa-9927-b4207e744223" path="/var/lib/kubelet/pods/e14e89e7-fb2e-44fa-9927-b4207e744223/volumes" Jul 14 19:55:35.356462 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:35.356381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:42.579769 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:42.579728 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:42.680595 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:42.680558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:43.108183 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.108151 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:43.150177 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.150144 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt"] Jul 14 19:55:43.150484 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.150471 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14e89e7-fb2e-44fa-9927-b4207e744223" containerName="manager" Jul 14 19:55:43.150528 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.150485 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14e89e7-fb2e-44fa-9927-b4207e744223" containerName="manager" Jul 14 19:55:43.150561 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.150540 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14e89e7-fb2e-44fa-9927-b4207e744223" containerName="manager" Jul 14 19:55:43.159412 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.159382 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.163479 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.163452 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 14 19:55:43.163634 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.163487 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rnm52\"" Jul 14 19:55:43.165539 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.165516 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt"] Jul 14 19:55:43.245789 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.245745 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31eea769-ff9f-455b-a656-01392a686861-openshift-service-ca\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.245789 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.245791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31eea769-ff9f-455b-a656-01392a686861-maas-api-tls\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.246001 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.245863 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jk9\" (UniqueName: \"kubernetes.io/projected/31eea769-ff9f-455b-a656-01392a686861-kube-api-access-l9jk9\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.346473 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.346432 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31eea769-ff9f-455b-a656-01392a686861-openshift-service-ca\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.346473 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.346472 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31eea769-ff9f-455b-a656-01392a686861-maas-api-tls\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.346686 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.346543 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jk9\" (UniqueName: \"kubernetes.io/projected/31eea769-ff9f-455b-a656-01392a686861-kube-api-access-l9jk9\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.346686 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:55:43.346650 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 14 19:55:43.346768 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:55:43.346728 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31eea769-ff9f-455b-a656-01392a686861-maas-api-tls podName:31eea769-ff9f-455b-a656-01392a686861 nodeName:}" failed. No retries permitted until 2026-07-14 19:55:43.846709291 +0000 UTC m=+549.295335725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31eea769-ff9f-455b-a656-01392a686861-maas-api-tls") pod "maas-api-6cf85b6845-tcngt" (UID: "31eea769-ff9f-455b-a656-01392a686861") : secret "maas-api-serving-cert" not found Jul 14 19:55:43.347210 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.347190 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31eea769-ff9f-455b-a656-01392a686861-openshift-service-ca\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.355572 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.355550 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jk9\" (UniqueName: \"kubernetes.io/projected/31eea769-ff9f-455b-a656-01392a686861-kube-api-access-l9jk9\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.440725 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.440691 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:43.603583 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.603551 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:43.692476 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.692392 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:43.851707 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.851673 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31eea769-ff9f-455b-a656-01392a686861-maas-api-tls\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:43.854219 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:43.854195 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31eea769-ff9f-455b-a656-01392a686861-maas-api-tls\") pod \"maas-api-6cf85b6845-tcngt\" (UID: \"31eea769-ff9f-455b-a656-01392a686861\") " pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:44.071560 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:44.071468 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:44.206129 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:44.206101 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt"] Jul 14 19:55:44.207692 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:55:44.207660 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31eea769_ff9f_455b_a656_01392a686861.slice/crio-a89b25a001c9870dd44705d30fda25180aad05a0f65cc9aea173f637e23bd104 WatchSource:0}: Error finding container a89b25a001c9870dd44705d30fda25180aad05a0f65cc9aea173f637e23bd104: Status 404 returned error can't find the container with id a89b25a001c9870dd44705d30fda25180aad05a0f65cc9aea173f637e23bd104 Jul 14 19:55:44.614424 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:44.614378 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" event={"ID":"31eea769-ff9f-455b-a656-01392a686861","Type":"ContainerStarted","Data":"a89b25a001c9870dd44705d30fda25180aad05a0f65cc9aea173f637e23bd104"} Jul 14 19:55:46.625602 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:46.625566 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" event={"ID":"31eea769-ff9f-455b-a656-01392a686861","Type":"ContainerStarted","Data":"ea436f3ac013aa74889e5fd6b7347a9c9eeccaae5d3eb296687ade99909658cd"} Jul 14 19:55:46.625985 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:46.625698 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:46.643650 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:46.643590 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" podStartSLOduration=1.734592969 podStartE2EDuration="3.643570361s" totalImagesPullingTime="1.908977392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:46.643446024 +0000 UTC m=+552.092072469" watchObservedRunningTime="2026-07-14 19:55:46.643570361 +0000 UTC m=+552.092196808" Jul 14 19:55:52.635304 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:52.635272 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cf85b6845-tcngt" Jul 14 19:55:53.228355 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:53.228324 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:53.291149 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:53.291116 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:53.603412 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:53.603334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:53.699798 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:55:53.699768 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:17.994094 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:17.993998 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8"] Jul 14 19:56:17.997608 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:17.997587 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.000875 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.000854 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wjq24\"" Jul 14 19:56:18.000875 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.000869 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 14 19:56:18.001173 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.000893 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 19:56:18.001173 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.000901 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 19:56:18.007283 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.007254 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8"] Jul 14 19:56:18.022288 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.022255 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:18.049521 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.049487 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.049521 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.049521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.049761 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.049541 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fca802-ad0c-4c70-96c4-8949bdca550d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.049761 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.049655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.049761 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.049702 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcmr\" (UniqueName: \"kubernetes.io/projected/62fca802-ad0c-4c70-96c4-8949bdca550d-kube-api-access-6pcmr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.049761 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.049754 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151123 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151057 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151214 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151297 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151250 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fca802-ad0c-4c70-96c4-8949bdca550d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151415 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151312 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151415 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151339 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcmr\" (UniqueName: \"kubernetes.io/projected/62fca802-ad0c-4c70-96c4-8949bdca550d-kube-api-access-6pcmr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151415 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151397 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151577 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151474 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151577 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151561 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.151667 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.151639 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.153568 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.153536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fca802-ad0c-4c70-96c4-8949bdca550d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.153780 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.153764 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fca802-ad0c-4c70-96c4-8949bdca550d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.162017 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.161985 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcmr\" (UniqueName: \"kubernetes.io/projected/62fca802-ad0c-4c70-96c4-8949bdca550d-kube-api-access-6pcmr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8\" (UID: \"62fca802-ad0c-4c70-96c4-8949bdca550d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.309147 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.309028 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:18.441781 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.441752 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8"] Jul 14 19:56:18.443710 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:56:18.443676 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fca802_ad0c_4c70_96c4_8949bdca550d.slice/crio-62ae383f70c18bce8872efea0d012df6aca1e15056d6b630e0a0159bbafb7375 WatchSource:0}: Error finding container 62ae383f70c18bce8872efea0d012df6aca1e15056d6b630e0a0159bbafb7375: Status 404 returned error can't find the container with id 62ae383f70c18bce8872efea0d012df6aca1e15056d6b630e0a0159bbafb7375 Jul 14 19:56:18.786413 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:18.786375 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" event={"ID":"62fca802-ad0c-4c70-96c4-8949bdca550d","Type":"ContainerStarted","Data":"62ae383f70c18bce8872efea0d012df6aca1e15056d6b630e0a0159bbafb7375"} Jul 14 19:56:19.793167 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:19.793126 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" event={"ID":"62fca802-ad0c-4c70-96c4-8949bdca550d","Type":"ContainerStarted","Data":"a599d07617ac795db1a514ad90f7ea0fb2c92683ed15ce962aa60ffdd5a063d8"} Jul 14 19:56:19.813415 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:19.813362 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" podStartSLOduration=1.584805154 podStartE2EDuration="2.813346402s" totalImagesPullingTime="1.228541248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:19.811888132 +0000 UTC m=+585.260514581" watchObservedRunningTime="2026-07-14 19:56:19.813346402 +0000 UTC m=+585.261972847" Jul 14 19:56:25.410963 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.410925 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j"] Jul 14 19:56:25.414794 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.414769 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.417514 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.417492 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 19:56:25.427436 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.427408 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j"] Jul 14 19:56:25.514284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.514235 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.514284 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.514295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwdc\" (UniqueName: \"kubernetes.io/projected/ce838c27-a756-4c04-9833-67272b283daf-kube-api-access-5kwdc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.514530 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.514396 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.514530 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.514435 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.514530 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.514462 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.514530 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.514477 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce838c27-a756-4c04-9833-67272b283daf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.568539 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.568497 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:25.615298 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615262 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615493 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615312 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwdc\" (UniqueName: \"kubernetes.io/projected/ce838c27-a756-4c04-9833-67272b283daf-kube-api-access-5kwdc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615493 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615358 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615493 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615382 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615493 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615409 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615493 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615431 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce838c27-a756-4c04-9833-67272b283daf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615789 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615766 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615839 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615786 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.615877 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.615842 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.617682 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.617661 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce838c27-a756-4c04-9833-67272b283daf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.617845 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.617828 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce838c27-a756-4c04-9833-67272b283daf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.625288 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.625248 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwdc\" (UniqueName: \"kubernetes.io/projected/ce838c27-a756-4c04-9833-67272b283daf-kube-api-access-5kwdc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m54j\" (UID: \"ce838c27-a756-4c04-9833-67272b283daf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.724253 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.724142 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:25.858778 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:25.858753 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j"] Jul 14 19:56:25.860655 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:56:25.860627 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce838c27_a756_4c04_9833_67272b283daf.slice/crio-e4ff35b96c68c23813b9438356dd7611e1b042b08fe14a1a637880581df60c4a WatchSource:0}: Error finding container e4ff35b96c68c23813b9438356dd7611e1b042b08fe14a1a637880581df60c4a: Status 404 returned error can't find the container with id e4ff35b96c68c23813b9438356dd7611e1b042b08fe14a1a637880581df60c4a Jul 14 19:56:26.835695 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:26.835662 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" event={"ID":"ce838c27-a756-4c04-9833-67272b283daf","Type":"ContainerStarted","Data":"aa7cb9f783538604f37b1e2a3bc9756ef695740b83e3575fa4c53bd6870cdbc3"} Jul 14 19:56:26.836131 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:26.835712 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" event={"ID":"ce838c27-a756-4c04-9833-67272b283daf","Type":"ContainerStarted","Data":"e4ff35b96c68c23813b9438356dd7611e1b042b08fe14a1a637880581df60c4a"} Jul 14 19:56:26.855125 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:26.855048 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" podStartSLOduration=1.676601029 podStartE2EDuration="1.855023416s" totalImagesPullingTime="178.422387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:26.853926069 +0000 UTC m=+592.302552520" watchObservedRunningTime="2026-07-14 19:56:26.855023416 +0000 UTC m=+592.303649863" Jul 14 19:56:28.310202 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:28.310165 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:28.310566 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:28.310228 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:28.322323 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:28.322296 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:28.857575 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:28.857549 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8" Jul 14 19:56:29.629213 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:29.629182 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:34.924926 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:34.924897 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:56:34.925324 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:34.925101 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 19:56:35.724990 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:35.724949 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:35.725214 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:35.725007 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:35.737278 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:35.737251 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:35.895832 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:35.895804 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m54j" Jul 14 19:56:36.535701 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:36.535667 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:53.993791 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:56:53.993761 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:02.470245 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:57:02.470146 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:04.137121 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:57:04.137077 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:30.556411 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:57:30.556372 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:25.273372 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:25.273334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:25.376971 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:25.376941 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:26.397340 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.397305 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-869cf89489-m7dx2"] Jul 14 19:58:26.397745 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.397554 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-869cf89489-m7dx2" podUID="621eefc2-1860-43de-bb4f-9a7e748215ae" containerName="authorino" containerID="cri-o://da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc" gracePeriod=30 Jul 14 19:58:26.449978 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.449952 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:26.550385 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.550346 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:26.644336 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.644310 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:58:26.806811 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.806712 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/621eefc2-1860-43de-bb4f-9a7e748215ae-tls-cert\") pod \"621eefc2-1860-43de-bb4f-9a7e748215ae\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " Jul 14 19:58:26.806811 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.806790 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29b6s\" (UniqueName: \"kubernetes.io/projected/621eefc2-1860-43de-bb4f-9a7e748215ae-kube-api-access-29b6s\") pod \"621eefc2-1860-43de-bb4f-9a7e748215ae\" (UID: \"621eefc2-1860-43de-bb4f-9a7e748215ae\") " Jul 14 19:58:26.808801 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.808768 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621eefc2-1860-43de-bb4f-9a7e748215ae-kube-api-access-29b6s" (OuterVolumeSpecName: "kube-api-access-29b6s") pod "621eefc2-1860-43de-bb4f-9a7e748215ae" (UID: "621eefc2-1860-43de-bb4f-9a7e748215ae"). InnerVolumeSpecName "kube-api-access-29b6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:58:26.816738 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.816709 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621eefc2-1860-43de-bb4f-9a7e748215ae-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "621eefc2-1860-43de-bb4f-9a7e748215ae" (UID: "621eefc2-1860-43de-bb4f-9a7e748215ae"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:58:26.908072 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.908024 2553 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/621eefc2-1860-43de-bb4f-9a7e748215ae-tls-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:58:26.908239 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:26.908090 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29b6s\" (UniqueName: \"kubernetes.io/projected/621eefc2-1860-43de-bb4f-9a7e748215ae-kube-api-access-29b6s\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:58:27.455008 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.454976 2553 generic.go:356] "Generic (PLEG): container finished" podID="621eefc2-1860-43de-bb4f-9a7e748215ae" containerID="da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc" exitCode=0 Jul 14 19:58:27.455483 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.455015 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869cf89489-m7dx2" Jul 14 19:58:27.455483 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.455090 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-869cf89489-m7dx2" event={"ID":"621eefc2-1860-43de-bb4f-9a7e748215ae","Type":"ContainerDied","Data":"da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc"} Jul 14 19:58:27.455483 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.455142 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-869cf89489-m7dx2" event={"ID":"621eefc2-1860-43de-bb4f-9a7e748215ae","Type":"ContainerDied","Data":"68d8850cc239d319fda7b4a8071ee7183ccb50f249def639fbbdb76c4f82c45a"} Jul 14 19:58:27.455483 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.455167 2553 scope.go:117] "RemoveContainer" containerID="da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc" Jul 14 19:58:27.466281 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.465093 2553 scope.go:117] "RemoveContainer" containerID="da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc" Jul 14 19:58:27.466560 ip-10-0-140-43 kubenswrapper[2553]: E0714 19:58:27.466538 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc\": container with ID starting with da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc not found: ID does not exist" containerID="da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc" Jul 14 19:58:27.466675 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.466565 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc"} err="failed to get container status \"da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc\": rpc error: code = NotFound desc = could not find container \"da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc\": container with ID starting with da68635f711b9973ab801672912ec8dd7bf4b72554d21e71b1c5f1370f8d9ddc not found: ID does not exist" Jul 14 19:58:27.472556 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.472533 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-869cf89489-m7dx2"] Jul 14 19:58:27.476378 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:27.476355 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-869cf89489-m7dx2"] Jul 14 19:58:29.037294 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:58:29.037263 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="621eefc2-1860-43de-bb4f-9a7e748215ae" path="/var/lib/kubelet/pods/621eefc2-1860-43de-bb4f-9a7e748215ae/volumes" Jul 14 19:59:13.219456 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:13.219418 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:13.320378 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:13.320344 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:39.467579 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.467521 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:39.572673 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.572630 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fc5448547-s87x6"] Jul 14 19:59:39.572914 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.572883 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fc5448547-s87x6" podUID="5772c7b9-298f-4c86-b564-1690a321ce87" containerName="manager" containerID="cri-o://2024ff0d92ffee84897abb7be4295af5f4a653a422401ffea77427fba8f1d341" gracePeriod=30 Jul 14 19:59:39.809799 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.809769 2553 generic.go:356] "Generic (PLEG): container finished" podID="5772c7b9-298f-4c86-b564-1690a321ce87" containerID="2024ff0d92ffee84897abb7be4295af5f4a653a422401ffea77427fba8f1d341" exitCode=0 Jul 14 19:59:39.809936 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.809826 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fc5448547-s87x6" event={"ID":"5772c7b9-298f-4c86-b564-1690a321ce87","Type":"ContainerDied","Data":"2024ff0d92ffee84897abb7be4295af5f4a653a422401ffea77427fba8f1d341"} Jul 14 19:59:39.809936 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.809863 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fc5448547-s87x6" event={"ID":"5772c7b9-298f-4c86-b564-1690a321ce87","Type":"ContainerDied","Data":"fc0b26155082fcd48e67436cc1f8e97f72c569e33e6d985d87e88faef36f595b"} Jul 14 19:59:39.809936 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.809881 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0b26155082fcd48e67436cc1f8e97f72c569e33e6d985d87e88faef36f595b" Jul 14 19:59:39.821321 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.821294 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:59:39.958836 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.958801 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772c7b9-298f-4c86-b564-1690a321ce87-webhook-cert\") pod \"5772c7b9-298f-4c86-b564-1690a321ce87\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " Jul 14 19:59:39.959104 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.958912 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk4dh\" (UniqueName: \"kubernetes.io/projected/5772c7b9-298f-4c86-b564-1690a321ce87-kube-api-access-bk4dh\") pod \"5772c7b9-298f-4c86-b564-1690a321ce87\" (UID: \"5772c7b9-298f-4c86-b564-1690a321ce87\") " Jul 14 19:59:39.960976 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.960950 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772c7b9-298f-4c86-b564-1690a321ce87-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5772c7b9-298f-4c86-b564-1690a321ce87" (UID: "5772c7b9-298f-4c86-b564-1690a321ce87"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:59:39.961102 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:39.960981 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5772c7b9-298f-4c86-b564-1690a321ce87-kube-api-access-bk4dh" (OuterVolumeSpecName: "kube-api-access-bk4dh") pod "5772c7b9-298f-4c86-b564-1690a321ce87" (UID: "5772c7b9-298f-4c86-b564-1690a321ce87"). InnerVolumeSpecName "kube-api-access-bk4dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:40.059613 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:40.059516 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772c7b9-298f-4c86-b564-1690a321ce87-webhook-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:59:40.059613 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:40.059552 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk4dh\" (UniqueName: \"kubernetes.io/projected/5772c7b9-298f-4c86-b564-1690a321ce87-kube-api-access-bk4dh\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 19:59:40.815379 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:40.815349 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fc5448547-s87x6" Jul 14 19:59:40.838684 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:40.838656 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fc5448547-s87x6"] Jul 14 19:59:40.844006 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:40.843984 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fc5448547-s87x6"] Jul 14 19:59:41.037832 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.037784 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5772c7b9-298f-4c86-b564-1690a321ce87" path="/var/lib/kubelet/pods/5772c7b9-298f-4c86-b564-1690a321ce87/volumes" Jul 14 19:59:41.594254 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.594222 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:41.685622 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.685589 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fc5448547-dfqhm"] Jul 14 19:59:41.686085 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.686044 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621eefc2-1860-43de-bb4f-9a7e748215ae" containerName="authorino" Jul 14 19:59:41.686085 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.686076 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="621eefc2-1860-43de-bb4f-9a7e748215ae" containerName="authorino" Jul 14 19:59:41.686085 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.686087 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772c7b9-298f-4c86-b564-1690a321ce87" containerName="manager" Jul 14 19:59:41.686207 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.686093 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772c7b9-298f-4c86-b564-1690a321ce87" containerName="manager" Jul 14 19:59:41.686207 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.686173 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="621eefc2-1860-43de-bb4f-9a7e748215ae" containerName="authorino" Jul 14 19:59:41.686207 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.686183 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772c7b9-298f-4c86-b564-1690a321ce87" containerName="manager" Jul 14 19:59:41.689273 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.689255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:41.692802 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.692778 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 19:59:41.692802 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.692789 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 19:59:41.692802 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.692781 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-llx58\"" Jul 14 19:59:41.697784 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.697760 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fc5448547-dfqhm"] Jul 14 19:59:41.775168 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.775138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmjc\" (UniqueName: \"kubernetes.io/projected/d28c1cba-dd0c-431c-9b00-3cae115cfc21-kube-api-access-fgmjc\") pod \"maas-controller-fc5448547-dfqhm\" (UID: \"d28c1cba-dd0c-431c-9b00-3cae115cfc21\") " pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:41.775332 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.775185 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d28c1cba-dd0c-431c-9b00-3cae115cfc21-webhook-cert\") pod \"maas-controller-fc5448547-dfqhm\" (UID: \"d28c1cba-dd0c-431c-9b00-3cae115cfc21\") " pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:41.876625 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.876593 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmjc\" (UniqueName: \"kubernetes.io/projected/d28c1cba-dd0c-431c-9b00-3cae115cfc21-kube-api-access-fgmjc\") pod \"maas-controller-fc5448547-dfqhm\" (UID: \"d28c1cba-dd0c-431c-9b00-3cae115cfc21\") " pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:41.876991 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.876649 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d28c1cba-dd0c-431c-9b00-3cae115cfc21-webhook-cert\") pod \"maas-controller-fc5448547-dfqhm\" (UID: \"d28c1cba-dd0c-431c-9b00-3cae115cfc21\") " pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:41.879163 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.879141 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d28c1cba-dd0c-431c-9b00-3cae115cfc21-webhook-cert\") pod \"maas-controller-fc5448547-dfqhm\" (UID: \"d28c1cba-dd0c-431c-9b00-3cae115cfc21\") " pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:41.885552 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:41.885523 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmjc\" (UniqueName: \"kubernetes.io/projected/d28c1cba-dd0c-431c-9b00-3cae115cfc21-kube-api-access-fgmjc\") pod \"maas-controller-fc5448547-dfqhm\" (UID: \"d28c1cba-dd0c-431c-9b00-3cae115cfc21\") " pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:42.000272 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.000235 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:42.133839 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.133808 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fc5448547-dfqhm"] Jul 14 19:59:42.137581 ip-10-0-140-43 kubenswrapper[2553]: W0714 19:59:42.137550 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28c1cba_dd0c_431c_9b00_3cae115cfc21.slice/crio-72a4bae3ecfc8d30eb458b0bbb5dce4c40bda34234fde708997981c118175176 WatchSource:0}: Error finding container 72a4bae3ecfc8d30eb458b0bbb5dce4c40bda34234fde708997981c118175176: Status 404 returned error can't find the container with id 72a4bae3ecfc8d30eb458b0bbb5dce4c40bda34234fde708997981c118175176 Jul 14 19:59:42.138961 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.138943 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:59:42.826495 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.826459 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fc5448547-dfqhm" event={"ID":"d28c1cba-dd0c-431c-9b00-3cae115cfc21","Type":"ContainerStarted","Data":"8a7c6323ebc407fc6982b178f278be481812ace5d6bb215cbdb8ab0ebd0dffb9"} Jul 14 19:59:42.826688 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.826524 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 19:59:42.826688 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.826543 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fc5448547-dfqhm" event={"ID":"d28c1cba-dd0c-431c-9b00-3cae115cfc21","Type":"ContainerStarted","Data":"72a4bae3ecfc8d30eb458b0bbb5dce4c40bda34234fde708997981c118175176"} Jul 14 19:59:42.845089 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:42.845019 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fc5448547-dfqhm" podStartSLOduration=1.329179995 podStartE2EDuration="1.84500598s" totalImagesPullingTime="515.825985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:59:42.844341278 +0000 UTC m=+788.292967738" watchObservedRunningTime="2026-07-14 19:59:42.84500598 +0000 UTC m=+788.293632425" Jul 14 19:59:47.947214 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:47.947183 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:48.047972 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:48.047935 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:53.833965 ip-10-0-140-43 kubenswrapper[2553]: I0714 19:59:53.833926 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fc5448547-dfqhm" Jul 14 20:00:00.137143 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.137053 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw"] Jul 14 20:00:00.140342 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.140327 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" Jul 14 20:00:00.142849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.142828 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:00:00.149861 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.149837 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw"] Jul 14 20:00:00.151139 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.151116 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" Jul 14 20:00:00.287959 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.287931 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw"] Jul 14 20:00:00.289958 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:00:00.289931 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566b7644_e4e0_4f42_bceb_b6921d984ca5.slice/crio-e58a4190f90064aad7b654ca94ef0fb5d903b359f3b7e39e7992c340e22a6b3e WatchSource:0}: Error finding container e58a4190f90064aad7b654ca94ef0fb5d903b359f3b7e39e7992c340e22a6b3e: Status 404 returned error can't find the container with id e58a4190f90064aad7b654ca94ef0fb5d903b359f3b7e39e7992c340e22a6b3e Jul 14 20:00:00.918172 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:00.918140 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" event={"ID":"566b7644-e4e0-4f42-bceb-b6921d984ca5","Type":"ContainerStarted","Data":"e58a4190f90064aad7b654ca94ef0fb5d903b359f3b7e39e7992c340e22a6b3e"} Jul 14 20:00:05.949045 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:05.949006 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" event={"ID":"566b7644-e4e0-4f42-bceb-b6921d984ca5","Type":"ContainerStarted","Data":"0e7ccd5c76963c72b5d26c720c3f8f1390fd7ccebb0196d08061ec90d97e5041"} Jul 14 20:00:05.965277 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:05.965230 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" podStartSLOduration=1.175148232 podStartE2EDuration="5.965214602s" totalImagesPullingTime="4.79006637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:00:05.963675023 +0000 UTC m=+811.412301481" watchObservedRunningTime="2026-07-14 20:00:05.965214602 +0000 UTC m=+811.413841048" Jul 14 20:00:06.955653 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:06.955618 2553 generic.go:356] "Generic (PLEG): container finished" podID="566b7644-e4e0-4f42-bceb-b6921d984ca5" containerID="0e7ccd5c76963c72b5d26c720c3f8f1390fd7ccebb0196d08061ec90d97e5041" exitCode=0 Jul 14 20:00:06.956014 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:06.955702 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" event={"ID":"566b7644-e4e0-4f42-bceb-b6921d984ca5","Type":"ContainerDied","Data":"0e7ccd5c76963c72b5d26c720c3f8f1390fd7ccebb0196d08061ec90d97e5041"} Jul 14 20:00:08.089913 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:08.089887 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" Jul 14 20:00:08.967033 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:08.967000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" event={"ID":"566b7644-e4e0-4f42-bceb-b6921d984ca5","Type":"ContainerDied","Data":"e58a4190f90064aad7b654ca94ef0fb5d903b359f3b7e39e7992c340e22a6b3e"} Jul 14 20:00:08.967223 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:08.967051 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e58a4190f90064aad7b654ca94ef0fb5d903b359f3b7e39e7992c340e22a6b3e" Jul 14 20:00:08.967223 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:08.967014 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-sfkdw" Jul 14 20:00:20.188234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:20.188197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:20.288347 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:20.288313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:20.405417 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:20.405380 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:26.649617 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:26.649586 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:26.756542 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:26.756513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:26.850541 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:26.850501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:26.984486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:26.984403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:35.537677 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:35.537645 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:35.602857 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:35.602830 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:46.529694 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:46.529658 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:46.628911 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:00:46.628877 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:01:34.964879 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:01:34.964799 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:01:34.969400 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:01:34.969380 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:01:34.984731 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:01:34.984705 2553 scope.go:117] "RemoveContainer" containerID="2024ff0d92ffee84897abb7be4295af5f4a653a422401ffea77427fba8f1d341" Jul 14 20:03:19.220890 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:03:19.220840 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:06:06.813753 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:06:06.813660 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:06:34.997797 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:06:34.997765 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:06:35.002257 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:06:35.002236 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:07:00.788257 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:07:00.788196 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:07:00.849876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:07:00.849839 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:08:48.284154 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:08:48.284117 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:11:35.032823 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:11:35.032794 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:11:35.035751 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:11:35.035719 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:11:37.469166 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:11:37.469136 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:11:44.236453 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:11:44.236411 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:11:52.499460 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:11:52.499430 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:11:57.631676 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:11:57.631588 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:00.764409 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:00.764368 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:00.866494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:00.866450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:01.237831 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:01.237799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:07.157107 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.157052 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:07.220159 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.220125 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:07.345195 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.345164 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp"] Jul 14 20:12:07.345550 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.345536 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566b7644-e4e0-4f42-bceb-b6921d984ca5" containerName="cleanup" Jul 14 20:12:07.345550 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.345551 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="566b7644-e4e0-4f42-bceb-b6921d984ca5" containerName="cleanup" Jul 14 20:12:07.345687 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.345618 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="566b7644-e4e0-4f42-bceb-b6921d984ca5" containerName="cleanup" Jul 14 20:12:07.348709 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.348685 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.352271 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.352246 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ef48f8e1-openshift-default-dockercfg-d6xzj\"" Jul 14 20:12:07.370767 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.370738 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp"] Jul 14 20:12:07.388998 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.388955 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:07.446109 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.445990 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-certs\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446109 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446034 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-podinfo\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446109 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446055 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-envoy\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446156 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-data\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446189 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-token\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446228 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-socket\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446244 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-credential-socket\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446260 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtmjx\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-kube-api-access-xtmjx\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.446321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.446305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b7606-bb6c-432e-9f17-c4677f860dff-istiod-ca-cert\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.484341 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.484308 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:07.547679 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547618 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-socket\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547679 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547681 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-credential-socket\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547842 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtmjx\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-kube-api-access-xtmjx\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547842 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547772 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b7606-bb6c-432e-9f17-c4677f860dff-istiod-ca-cert\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547842 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-certs\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547997 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547849 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-podinfo\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547997 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-envoy\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547997 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547918 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-data\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.547997 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.547945 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-token\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.548302 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.548123 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-credential-socket\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.548302 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.548146 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-socket\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.548488 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.548466 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-data\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.548549 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.548485 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-certs\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.548549 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.548527 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b7606-bb6c-432e-9f17-c4677f860dff-istiod-ca-cert\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.550206 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.550186 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-envoy\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.550358 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.550338 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-podinfo\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.559463 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.559429 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-token\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.559921 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.559895 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtmjx\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-kube-api-access-xtmjx\") pod \"e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.661364 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.661319 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:07.803996 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.803965 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp"] Jul 14 20:12:07.805517 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:12:07.805492 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7b7606_bb6c_432e_9f17_c4677f860dff.slice/crio-e55c38368ea9b6b22fadf1b9349d7cda3f1967c9c57aca55697a807a3a025011 WatchSource:0}: Error finding container e55c38368ea9b6b22fadf1b9349d7cda3f1967c9c57aca55697a807a3a025011: Status 404 returned error can't find the container with id e55c38368ea9b6b22fadf1b9349d7cda3f1967c9c57aca55697a807a3a025011 Jul 14 20:12:07.807335 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:07.807318 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:12:08.634780 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.634743 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" event={"ID":"6a7b7606-bb6c-432e-9f17-c4677f860dff","Type":"ContainerStarted","Data":"e55c38368ea9b6b22fadf1b9349d7cda3f1967c9c57aca55697a807a3a025011"} Jul 14 20:12:08.678087 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.678015 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:08.733292 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.733259 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws"] Jul 14 20:12:08.737984 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.737957 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.740484 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.740455 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ef48f8e1-serving-cert\"" Jul 14 20:12:08.749377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.749346 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws"] Jul 14 20:12:08.760817 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.760776 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5272a5c5-cb10-4297-b979-2653174f5dc9-openshift-service-ca\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.761157 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.761134 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.761283 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.761205 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tglkf\" (UniqueName: \"kubernetes.io/projected/5272a5c5-cb10-4297-b979-2653174f5dc9-kube-api-access-tglkf\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.770791 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.770755 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:08.846172 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.846136 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:08.862456 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.862419 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tglkf\" (UniqueName: \"kubernetes.io/projected/5272a5c5-cb10-4297-b979-2653174f5dc9-kube-api-access-tglkf\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.862609 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.862526 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5272a5c5-cb10-4297-b979-2653174f5dc9-openshift-service-ca\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.862609 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.862576 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.862741 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:08.862715 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-serving-cert: secret "maas-api-e2e-ait-ef48f8e1-serving-cert" not found Jul 14 20:12:08.862811 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:08.862799 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls podName:5272a5c5-cb10-4297-b979-2653174f5dc9 nodeName:}" failed. No retries permitted until 2026-07-14 20:12:09.362776193 +0000 UTC m=+1534.811402629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls") pod "maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" (UID: "5272a5c5-cb10-4297-b979-2653174f5dc9") : secret "maas-api-e2e-ait-ef48f8e1-serving-cert" not found Jul 14 20:12:08.863234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.863213 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5272a5c5-cb10-4297-b979-2653174f5dc9-openshift-service-ca\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:08.872805 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:08.872780 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tglkf\" (UniqueName: \"kubernetes.io/projected/5272a5c5-cb10-4297-b979-2653174f5dc9-kube-api-access-tglkf\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:09.002991 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:09.002905 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:09.367905 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:09.367818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:09.370683 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:09.370653 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls\") pod \"maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:09.651816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:09.651780 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:09.816173 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:09.816137 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws"] Jul 14 20:12:09.819514 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:12:09.819478 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5272a5c5_cb10_4297_b979_2653174f5dc9.slice/crio-7a527a5b4ca4bc4d574fae5ebe568187c2632665bc61a88168106f62c4d4e8d6 WatchSource:0}: Error finding container 7a527a5b4ca4bc4d574fae5ebe568187c2632665bc61a88168106f62c4d4e8d6: Status 404 returned error can't find the container with id 7a527a5b4ca4bc4d574fae5ebe568187c2632665bc61a88168106f62c4d4e8d6 Jul 14 20:12:10.646696 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:10.646656 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:10.648825 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:10.648788 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" event={"ID":"5272a5c5-cb10-4297-b979-2653174f5dc9","Type":"ContainerStarted","Data":"7a527a5b4ca4bc4d574fae5ebe568187c2632665bc61a88168106f62c4d4e8d6"} Jul 14 20:12:10.746174 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:10.746139 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:10.927545 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:10.927510 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:11.292479 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.292445 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:11.292543 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.292518 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:11.292591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.292562 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:11.657872 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.657833 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" event={"ID":"6a7b7606-bb6c-432e-9f17-c4677f860dff","Type":"ContainerStarted","Data":"cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3"} Jul 14 20:12:11.659337 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.659309 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" event={"ID":"5272a5c5-cb10-4297-b979-2653174f5dc9","Type":"ContainerStarted","Data":"b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15"} Jul 14 20:12:11.659449 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.659363 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:11.661660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.661639 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:11.680197 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.680146 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podStartSLOduration=1.195407593 podStartE2EDuration="4.680128737s" totalImagesPullingTime="3.484721144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:11.679325356 +0000 UTC m=+1537.127951850" watchObservedRunningTime="2026-07-14 20:12:11.680128737 +0000 UTC m=+1537.128755183" Jul 14 20:12:11.699533 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:11.699483 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" podStartSLOduration=2.229629838 podStartE2EDuration="3.699466946s" totalImagesPullingTime="1.469837108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:11.697506717 +0000 UTC m=+1537.146133187" watchObservedRunningTime="2026-07-14 20:12:11.699466946 +0000 UTC m=+1537.148093392" Jul 14 20:12:13.666410 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:13.666356 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:12:13.667110 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:13.667047 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:12:14.668214 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:14.668172 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:12:14.668850 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:14.668815 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:12:15.670331 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:15.670292 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:12:15.670954 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:15.670921 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:12:16.672607 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:16.672479 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:12:16.672607 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:16.672566 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:12:17.670358 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:17.670330 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:17.713280 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:17.713238 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:12:17.713739 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:17.713326 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:12:18.754312 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:18.754248 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:12:18.754792 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:18.754350 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:12:19.485182 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.484426 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:37856->10.134.0.43:15021: read: connection reset by peer" start-of-body= Jul 14 20:12:19.485182 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.484497 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:37856->10.134.0.43:15021: read: connection reset by peer" Jul 14 20:12:19.662662 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.662569 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:12:19.662662 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.662627 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:12:19.712983 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.712949 2553 generic.go:356] "Generic (PLEG): container finished" podID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerID="cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3" exitCode=0 Jul 14 20:12:19.713186 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.713030 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" event={"ID":"6a7b7606-bb6c-432e-9f17-c4677f860dff","Type":"ContainerDied","Data":"cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3"} Jul 14 20:12:19.713477 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.713464 2553 scope.go:117] "RemoveContainer" containerID="cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3" Jul 14 20:12:19.715312 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.715285 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:19.715372 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.715350 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:19.715411 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:19.715383 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:20.661862 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:20.661830 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:20.719626 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:20.719594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" event={"ID":"6a7b7606-bb6c-432e-9f17-c4677f860dff","Type":"ContainerStarted","Data":"8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8"} Jul 14 20:12:21.397179 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:21.397143 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:21.500470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:21.500437 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:21.661943 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:21.661842 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:21.662408 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:21.662029 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:12:21.662408 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:21.662131 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:12:22.662203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:22.662153 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:22.662602 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:22.662483 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:12:22.662602 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:22.662534 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:12:23.661759 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:23.661727 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:12:23.661944 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:23.661786 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:12:24.125177 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.125050 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws"] Jul 14 20:12:24.125635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.125358 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" podUID="5272a5c5-cb10-4297-b979-2653174f5dc9" containerName="maas-api" containerID="cri-o://b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15" gracePeriod=30 Jul 14 20:12:24.151827 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.151796 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:24.409537 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.409498 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:24.517494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.517450 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls\") pod \"5272a5c5-cb10-4297-b979-2653174f5dc9\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " Jul 14 20:12:24.517680 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.517552 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5272a5c5-cb10-4297-b979-2653174f5dc9-openshift-service-ca\") pod \"5272a5c5-cb10-4297-b979-2653174f5dc9\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " Jul 14 20:12:24.517680 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.517595 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tglkf\" (UniqueName: \"kubernetes.io/projected/5272a5c5-cb10-4297-b979-2653174f5dc9-kube-api-access-tglkf\") pod \"5272a5c5-cb10-4297-b979-2653174f5dc9\" (UID: \"5272a5c5-cb10-4297-b979-2653174f5dc9\") " Jul 14 20:12:24.517946 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.517917 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5272a5c5-cb10-4297-b979-2653174f5dc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5272a5c5-cb10-4297-b979-2653174f5dc9" (UID: "5272a5c5-cb10-4297-b979-2653174f5dc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:12:24.519832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.519794 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5272a5c5-cb10-4297-b979-2653174f5dc9" (UID: "5272a5c5-cb10-4297-b979-2653174f5dc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:12:24.519832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.519804 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5272a5c5-cb10-4297-b979-2653174f5dc9-kube-api-access-tglkf" (OuterVolumeSpecName: "kube-api-access-tglkf") pod "5272a5c5-cb10-4297-b979-2653174f5dc9" (UID: "5272a5c5-cb10-4297-b979-2653174f5dc9"). InnerVolumeSpecName "kube-api-access-tglkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:12:24.618298 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.618257 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5272a5c5-cb10-4297-b979-2653174f5dc9-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:24.618298 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.618290 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tglkf\" (UniqueName: \"kubernetes.io/projected/5272a5c5-cb10-4297-b979-2653174f5dc9-kube-api-access-tglkf\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:24.618298 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.618300 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5272a5c5-cb10-4297-b979-2653174f5dc9-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:24.669029 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.668929 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:24.742744 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.742693 2553 generic.go:356] "Generic (PLEG): container finished" podID="5272a5c5-cb10-4297-b979-2653174f5dc9" containerID="b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15" exitCode=0 Jul 14 20:12:24.742941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.742760 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" Jul 14 20:12:24.742941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.742763 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" event={"ID":"5272a5c5-cb10-4297-b979-2653174f5dc9","Type":"ContainerDied","Data":"b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15"} Jul 14 20:12:24.742941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.742831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws" event={"ID":"5272a5c5-cb10-4297-b979-2653174f5dc9","Type":"ContainerDied","Data":"7a527a5b4ca4bc4d574fae5ebe568187c2632665bc61a88168106f62c4d4e8d6"} Jul 14 20:12:24.742941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.742855 2553 scope.go:117] "RemoveContainer" containerID="b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15" Jul 14 20:12:24.744602 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.744576 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:24.754075 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.754040 2553 scope.go:117] "RemoveContainer" containerID="b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15" Jul 14 20:12:24.754443 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:24.754424 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15\": container with ID starting with b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15 not found: ID does not exist" containerID="b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15" Jul 14 20:12:24.754494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.754454 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15"} err="failed to get container status \"b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15\": rpc error: code = NotFound desc = could not find container \"b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15\": container with ID starting with b87d90a1de1c1d785fe4d4729acd42cf5fe46cdddfc12e27e9efb5aaddaf1d15 not found: ID does not exist" Jul 14 20:12:24.783197 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.783160 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws"] Jul 14 20:12:24.792018 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:24.791985 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef48f8e1-6b4cd89d7c-qm9ws"] Jul 14 20:12:25.037429 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:25.037345 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5272a5c5-cb10-4297-b979-2653174f5dc9" path="/var/lib/kubelet/pods/5272a5c5-cb10-4297-b979-2653174f5dc9/volumes" Jul 14 20:12:28.666907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:28.666874 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:34.731806 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:34.731750 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:34.798770 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:34.798735 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:38.852436 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:38.852397 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp"] Jul 14 20:12:38.852896 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:38.852692 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" containerID="cri-o://8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8" gracePeriod=30 Jul 14 20:12:38.878531 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:38.878492 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:39.236718 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:39.236686 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:39.334106 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:39.334035 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:39.438641 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:39.438607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:39.745570 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:39.745536 2553 patch_prober.go:28] interesting pod/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:12:39.745750 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:39.745605 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:12:40.663029 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.662994 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:40.742877 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.742844 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp"] Jul 14 20:12:40.743472 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.743447 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5272a5c5-cb10-4297-b979-2653174f5dc9" containerName="maas-api" Jul 14 20:12:40.743472 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.743472 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5272a5c5-cb10-4297-b979-2653174f5dc9" containerName="maas-api" Jul 14 20:12:40.743651 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.743551 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5272a5c5-cb10-4297-b979-2653174f5dc9" containerName="maas-api" Jul 14 20:12:40.748289 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.748265 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.751031 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.751008 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f4a7e811-serving-cert\"" Jul 14 20:12:40.759005 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.758980 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp"] Jul 14 20:12:40.865658 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.865622 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808d5fda-088d-45f8-a3a4-b84f3a898e0e-maas-api-tls\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.865848 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.865692 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6bg\" (UniqueName: \"kubernetes.io/projected/808d5fda-088d-45f8-a3a4-b84f3a898e0e-kube-api-access-wv6bg\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.865848 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.865739 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808d5fda-088d-45f8-a3a4-b84f3a898e0e-openshift-service-ca\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.967093 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.966963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6bg\" (UniqueName: \"kubernetes.io/projected/808d5fda-088d-45f8-a3a4-b84f3a898e0e-kube-api-access-wv6bg\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.967093 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.967012 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808d5fda-088d-45f8-a3a4-b84f3a898e0e-openshift-service-ca\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.967093 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.967089 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808d5fda-088d-45f8-a3a4-b84f3a898e0e-maas-api-tls\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.967760 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.967727 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808d5fda-088d-45f8-a3a4-b84f3a898e0e-openshift-service-ca\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.969516 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.969486 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808d5fda-088d-45f8-a3a4-b84f3a898e0e-maas-api-tls\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:40.978643 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:40.978619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6bg\" (UniqueName: \"kubernetes.io/projected/808d5fda-088d-45f8-a3a4-b84f3a898e0e-kube-api-access-wv6bg\") pod \"maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:41.060809 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.060776 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:41.203049 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.203024 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp"] Jul 14 20:12:41.205232 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:12:41.205192 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808d5fda_088d_45f8_a3a4_b84f3a898e0e.slice/crio-01f19a28e25fafaf4e7ba3463823de015c68ab83e288f789d1465aea3acd3ecf WatchSource:0}: Error finding container 01f19a28e25fafaf4e7ba3463823de015c68ab83e288f789d1465aea3acd3ecf: Status 404 returned error can't find the container with id 01f19a28e25fafaf4e7ba3463823de015c68ab83e288f789d1465aea3acd3ecf Jul 14 20:12:41.437625 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.437574 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:41.536749 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.536714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:41.836030 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.835939 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" event={"ID":"808d5fda-088d-45f8-a3a4-b84f3a898e0e","Type":"ContainerStarted","Data":"fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9"} Jul 14 20:12:41.836030 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.835994 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" event={"ID":"808d5fda-088d-45f8-a3a4-b84f3a898e0e","Type":"ContainerStarted","Data":"01f19a28e25fafaf4e7ba3463823de015c68ab83e288f789d1465aea3acd3ecf"} Jul 14 20:12:41.836482 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.836079 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:41.854596 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:41.854495 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" podStartSLOduration=1.5065655470000001 podStartE2EDuration="1.854472113s" totalImagesPullingTime="347.906566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:41.854189893 +0000 UTC m=+1567.302816362" watchObservedRunningTime="2026-07-14 20:12:41.854472113 +0000 UTC m=+1567.303098560" Jul 14 20:12:44.112570 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.112499 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:44.198289 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198262 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b7606-bb6c-432e-9f17-c4677f860dff-istiod-ca-cert\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198310 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-envoy\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198340 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-certs\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198360 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-data\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198392 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-credential-socket\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198450 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtmjx\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-kube-api-access-xtmjx\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198484 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-podinfo\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198507 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-token\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198538 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-socket\") pod \"6a7b7606-bb6c-432e-9f17-c4677f860dff\" (UID: \"6a7b7606-bb6c-432e-9f17-c4677f860dff\") " Jul 14 20:12:44.198877 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198691 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:12:44.198877 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198741 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:12:44.198877 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198746 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7b7606-bb6c-432e-9f17-c4677f860dff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:12:44.199039 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198893 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:12:44.199039 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198970 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b7606-bb6c-432e-9f17-c4677f860dff-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.199039 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.198990 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.199039 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.199004 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.199039 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.199017 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.199333 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.199279 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:12:44.200891 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.200866 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:12:44.200992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.200865 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:12:44.200992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.200896 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-kube-api-access-xtmjx" (OuterVolumeSpecName: "kube-api-access-xtmjx") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "kube-api-access-xtmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:12:44.201459 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.201440 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a7b7606-bb6c-432e-9f17-c4677f860dff" (UID: "6a7b7606-bb6c-432e-9f17-c4677f860dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:12:44.300350 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.300300 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtmjx\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-kube-api-access-xtmjx\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.300350 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.300333 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.300350 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.300344 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.300350 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.300352 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.300350 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.300360 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7b7606-bb6c-432e-9f17-c4677f860dff-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:12:44.853151 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.853115 2553 generic.go:356] "Generic (PLEG): container finished" podID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerID="8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8" exitCode=0 Jul 14 20:12:44.853329 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.853170 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" Jul 14 20:12:44.853329 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.853197 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" event={"ID":"6a7b7606-bb6c-432e-9f17-c4677f860dff","Type":"ContainerDied","Data":"8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8"} Jul 14 20:12:44.853329 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.853249 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp" event={"ID":"6a7b7606-bb6c-432e-9f17-c4677f860dff","Type":"ContainerDied","Data":"e55c38368ea9b6b22fadf1b9349d7cda3f1967c9c57aca55697a807a3a025011"} Jul 14 20:12:44.853329 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.853270 2553 scope.go:117] "RemoveContainer" containerID="8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8" Jul 14 20:12:44.862569 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.862549 2553 scope.go:117] "RemoveContainer" containerID="cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3" Jul 14 20:12:44.871194 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.871175 2553 scope.go:117] "RemoveContainer" containerID="8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8" Jul 14 20:12:44.871502 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:44.871482 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8\": container with ID starting with 8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8 not found: ID does not exist" containerID="8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8" Jul 14 20:12:44.871550 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.871509 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8"} err="failed to get container status \"8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8\": rpc error: code = NotFound desc = could not find container \"8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8\": container with ID starting with 8007ffa27472a8c4330be015b0513ce173f3e9d9f45d017cfdbd4052ae657fd8 not found: ID does not exist" Jul 14 20:12:44.871550 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.871527 2553 scope.go:117] "RemoveContainer" containerID="cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3" Jul 14 20:12:44.871802 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:44.871785 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3\": container with ID starting with cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3 not found: ID does not exist" containerID="cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3" Jul 14 20:12:44.871848 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.871811 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3"} err="failed to get container status \"cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3\": rpc error: code = NotFound desc = could not find container \"cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3\": container with ID starting with cb226507af55c8bff57602f2234c4606dea8955e4afada843b5ad968ccb765b3 not found: ID does not exist" Jul 14 20:12:44.876984 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.876960 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp"] Jul 14 20:12:44.881272 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:44.881249 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ef48f8e1-openshift-default-6447d6c465-4ddzp"] Jul 14 20:12:45.036496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.036461 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" path="/var/lib/kubelet/pods/6a7b7606-bb6c-432e-9f17-c4677f860dff/volumes" Jul 14 20:12:45.130893 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.130855 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:45.169903 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.169862 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7"] Jul 14 20:12:45.170247 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.170233 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" Jul 14 20:12:45.170247 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.170247 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" Jul 14 20:12:45.170376 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.170317 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" Jul 14 20:12:45.174910 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.174880 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:12:45.178564 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.178025 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:12:45.180616 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.180590 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7"] Jul 14 20:12:45.194356 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.194324 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:45.310460 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.310418 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaba4270-934f-41c0-92f3-41aa3b900240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7\" (UID: \"aaba4270-934f-41c0-92f3-41aa3b900240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:12:45.410929 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.410886 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaba4270-934f-41c0-92f3-41aa3b900240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7\" (UID: \"aaba4270-934f-41c0-92f3-41aa3b900240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:12:45.411612 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.411589 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaba4270-934f-41c0-92f3-41aa3b900240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7\" (UID: \"aaba4270-934f-41c0-92f3-41aa3b900240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:12:45.488980 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.488935 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:12:45.635319 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.635288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7"] Jul 14 20:12:45.637452 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:12:45.637421 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaba4270_934f_41c0_92f3_41aa3b900240.slice/crio-cb04a098c2cdc0250d2c86cd2c220750fb2f6631836d0661bd55e063cc2e0d6e WatchSource:0}: Error finding container cb04a098c2cdc0250d2c86cd2c220750fb2f6631836d0661bd55e063cc2e0d6e: Status 404 returned error can't find the container with id cb04a098c2cdc0250d2c86cd2c220750fb2f6631836d0661bd55e063cc2e0d6e Jul 14 20:12:45.859591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:45.859553 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" event={"ID":"aaba4270-934f-41c0-92f3-41aa3b900240","Type":"ContainerStarted","Data":"cb04a098c2cdc0250d2c86cd2c220750fb2f6631836d0661bd55e063cc2e0d6e"} Jul 14 20:12:46.867465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:46.867428 2553 generic.go:356] "Generic (PLEG): container finished" podID="aaba4270-934f-41c0-92f3-41aa3b900240" containerID="f4aeb9e1b67aac30db18f9ece2e6909de6358fb8028f5e90d4843617b09a8be2" exitCode=7 Jul 14 20:12:46.867893 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:46.867510 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" event={"ID":"aaba4270-934f-41c0-92f3-41aa3b900240","Type":"ContainerDied","Data":"f4aeb9e1b67aac30db18f9ece2e6909de6358fb8028f5e90d4843617b09a8be2"} Jul 14 20:12:46.867893 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:46.867708 2553 scope.go:117] "RemoveContainer" containerID="f4aeb9e1b67aac30db18f9ece2e6909de6358fb8028f5e90d4843617b09a8be2" Jul 14 20:12:47.847156 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:47.847054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:12:47.874738 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:47.874701 2553 generic.go:356] "Generic (PLEG): container finished" podID="aaba4270-934f-41c0-92f3-41aa3b900240" containerID="1cba9b53b8a13a7f5b45777793ce810b30e58d6482d3f28cde4ddb2b244fcf15" exitCode=7 Jul 14 20:12:47.875252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:47.874796 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" event={"ID":"aaba4270-934f-41c0-92f3-41aa3b900240","Type":"ContainerDied","Data":"1cba9b53b8a13a7f5b45777793ce810b30e58d6482d3f28cde4ddb2b244fcf15"} Jul 14 20:12:47.875252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:47.874864 2553 scope.go:117] "RemoveContainer" containerID="f4aeb9e1b67aac30db18f9ece2e6909de6358fb8028f5e90d4843617b09a8be2" Jul 14 20:12:47.875252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:47.875039 2553 scope.go:117] "RemoveContainer" containerID="1cba9b53b8a13a7f5b45777793ce810b30e58d6482d3f28cde4ddb2b244fcf15" Jul 14 20:12:47.875392 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:47.875330 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7_odh-ai-gateway-infra(aaba4270-934f-41c0-92f3-41aa3b900240)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" Jul 14 20:12:48.803090 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:48.803031 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:12:48.882105 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:12:48.882080 2553 scope.go:117] "RemoveContainer" containerID="1cba9b53b8a13a7f5b45777793ce810b30e58d6482d3f28cde4ddb2b244fcf15" Jul 14 20:12:48.882477 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:12:48.882266 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7_odh-ai-gateway-infra(aaba4270-934f-41c0-92f3-41aa3b900240)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" Jul 14 20:13:01.275017 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:01.274984 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:01.333767 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:01.333734 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:04.030535 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:04.030502 2553 scope.go:117] "RemoveContainer" containerID="1cba9b53b8a13a7f5b45777793ce810b30e58d6482d3f28cde4ddb2b244fcf15" Jul 14 20:13:04.965721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:04.965628 2553 generic.go:356] "Generic (PLEG): container finished" podID="aaba4270-934f-41c0-92f3-41aa3b900240" containerID="dd5ee63067acecb08bb559feca9d5d4d15884c9c31dba9a462ac0ae5b1082ff6" exitCode=0 Jul 14 20:13:04.965721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:04.965701 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" event={"ID":"aaba4270-934f-41c0-92f3-41aa3b900240","Type":"ContainerDied","Data":"dd5ee63067acecb08bb559feca9d5d4d15884c9c31dba9a462ac0ae5b1082ff6"} Jul 14 20:13:04.965892 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:04.965755 2553 scope.go:117] "RemoveContainer" containerID="1cba9b53b8a13a7f5b45777793ce810b30e58d6482d3f28cde4ddb2b244fcf15" Jul 14 20:13:05.999078 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:05.999031 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7"] Jul 14 20:13:06.104275 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.104245 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:13:06.192990 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.192949 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaba4270-934f-41c0-92f3-41aa3b900240-maas-api-service-ca\") pod \"aaba4270-934f-41c0-92f3-41aa3b900240\" (UID: \"aaba4270-934f-41c0-92f3-41aa3b900240\") " Jul 14 20:13:06.193357 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.193334 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaba4270-934f-41c0-92f3-41aa3b900240-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aaba4270-934f-41c0-92f3-41aa3b900240" (UID: "aaba4270-934f-41c0-92f3-41aa3b900240"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:13:06.294486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.294393 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaba4270-934f-41c0-92f3-41aa3b900240-maas-api-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:13:06.978944 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.978906 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" event={"ID":"aaba4270-934f-41c0-92f3-41aa3b900240","Type":"ContainerDied","Data":"cb04a098c2cdc0250d2c86cd2c220750fb2f6631836d0661bd55e063cc2e0d6e"} Jul 14 20:13:06.979143 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.978962 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb04a098c2cdc0250d2c86cd2c220750fb2f6631836d0661bd55e063cc2e0d6e" Jul 14 20:13:06.979143 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:06.978922 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7" Jul 14 20:13:07.012077 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:07.012035 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7"] Jul 14 20:13:07.016097 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:07.016051 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4r8d7"] Jul 14 20:13:07.037055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:07.037027 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" path="/var/lib/kubelet/pods/aaba4270-934f-41c0-92f3-41aa3b900240/volumes" Jul 14 20:13:45.233991 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.233951 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt"] Jul 14 20:13:45.234518 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234499 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" Jul 14 20:13:45.234562 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234521 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" Jul 14 20:13:45.234562 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234537 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:45.234562 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234546 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:45.234562 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234559 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:45.234687 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234568 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:45.234687 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234649 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:45.234687 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234660 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:45.234687 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.234671 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7b7606-bb6c-432e-9f17-c4677f860dff" containerName="istio-proxy" Jul 14 20:13:45.237981 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.237964 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:45.240533 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.240509 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:13:45.244982 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.244953 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt"] Jul 14 20:13:45.320869 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.320833 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:45.348378 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.348337 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feb0822-1227-4a77-b917-bc6d057e5909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt\" (UID: \"4feb0822-1227-4a77-b917-bc6d057e5909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:45.429973 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.429937 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:45.450095 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.450029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feb0822-1227-4a77-b917-bc6d057e5909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt\" (UID: \"4feb0822-1227-4a77-b917-bc6d057e5909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:45.450730 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.450706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feb0822-1227-4a77-b917-bc6d057e5909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt\" (UID: \"4feb0822-1227-4a77-b917-bc6d057e5909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:45.552148 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.552028 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:45.690607 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:45.690570 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt"] Jul 14 20:13:45.692299 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:13:45.692271 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feb0822_1227_4a77_b917_bc6d057e5909.slice/crio-9f72cb45d94fb862f1559cc1abc142b42143684c329e6a3b8d2870626320ab1a WatchSource:0}: Error finding container 9f72cb45d94fb862f1559cc1abc142b42143684c329e6a3b8d2870626320ab1a: Status 404 returned error can't find the container with id 9f72cb45d94fb862f1559cc1abc142b42143684c329e6a3b8d2870626320ab1a Jul 14 20:13:46.182179 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:46.182077 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" event={"ID":"4feb0822-1227-4a77-b917-bc6d057e5909","Type":"ContainerStarted","Data":"b5d814a5de5e87803e2cac73f822d5adeabcf056b7573054a672b78b0f8f2ed0"} Jul 14 20:13:46.182179 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:46.182134 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" event={"ID":"4feb0822-1227-4a77-b917-bc6d057e5909","Type":"ContainerStarted","Data":"9f72cb45d94fb862f1559cc1abc142b42143684c329e6a3b8d2870626320ab1a"} Jul 14 20:13:46.199544 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:46.199490 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" podStartSLOduration=1.199470762 podStartE2EDuration="1.199470762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:13:46.19752768 +0000 UTC m=+1631.646154146" watchObservedRunningTime="2026-07-14 20:13:46.199470762 +0000 UTC m=+1631.648097207" Jul 14 20:13:47.188542 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:47.188500 2553 generic.go:356] "Generic (PLEG): container finished" podID="4feb0822-1227-4a77-b917-bc6d057e5909" containerID="b5d814a5de5e87803e2cac73f822d5adeabcf056b7573054a672b78b0f8f2ed0" exitCode=0 Jul 14 20:13:47.188934 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:47.188541 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" event={"ID":"4feb0822-1227-4a77-b917-bc6d057e5909","Type":"ContainerDied","Data":"b5d814a5de5e87803e2cac73f822d5adeabcf056b7573054a672b78b0f8f2ed0"} Jul 14 20:13:48.331549 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:48.331522 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:48.377440 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:48.377404 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feb0822-1227-4a77-b917-bc6d057e5909-maas-api-service-ca\") pod \"4feb0822-1227-4a77-b917-bc6d057e5909\" (UID: \"4feb0822-1227-4a77-b917-bc6d057e5909\") " Jul 14 20:13:48.377811 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:48.377788 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4feb0822-1227-4a77-b917-bc6d057e5909-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4feb0822-1227-4a77-b917-bc6d057e5909" (UID: "4feb0822-1227-4a77-b917-bc6d057e5909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:13:48.479057 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:48.478964 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feb0822-1227-4a77-b917-bc6d057e5909-maas-api-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:13:49.199870 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:49.199843 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" Jul 14 20:13:49.200040 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:49.199882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt" event={"ID":"4feb0822-1227-4a77-b917-bc6d057e5909","Type":"ContainerDied","Data":"9f72cb45d94fb862f1559cc1abc142b42143684c329e6a3b8d2870626320ab1a"} Jul 14 20:13:49.200040 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:49.199932 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f72cb45d94fb862f1559cc1abc142b42143684c329e6a3b8d2870626320ab1a" Jul 14 20:13:51.751791 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:51.751747 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp"] Jul 14 20:13:51.752335 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:51.752051 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" podUID="808d5fda-088d-45f8-a3a4-b84f3a898e0e" containerName="maas-api" containerID="cri-o://fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9" gracePeriod=30 Jul 14 20:13:51.789315 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:51.789257 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:52.011546 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.011476 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:13:52.108466 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.108425 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808d5fda-088d-45f8-a3a4-b84f3a898e0e-maas-api-tls\") pod \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " Jul 14 20:13:52.108672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.108488 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808d5fda-088d-45f8-a3a4-b84f3a898e0e-openshift-service-ca\") pod \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " Jul 14 20:13:52.108672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.108538 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv6bg\" (UniqueName: \"kubernetes.io/projected/808d5fda-088d-45f8-a3a4-b84f3a898e0e-kube-api-access-wv6bg\") pod \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\" (UID: \"808d5fda-088d-45f8-a3a4-b84f3a898e0e\") " Jul 14 20:13:52.108868 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.108841 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808d5fda-088d-45f8-a3a4-b84f3a898e0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "808d5fda-088d-45f8-a3a4-b84f3a898e0e" (UID: "808d5fda-088d-45f8-a3a4-b84f3a898e0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:13:52.110741 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.110709 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808d5fda-088d-45f8-a3a4-b84f3a898e0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "808d5fda-088d-45f8-a3a4-b84f3a898e0e" (UID: "808d5fda-088d-45f8-a3a4-b84f3a898e0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:13:52.110741 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.110712 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808d5fda-088d-45f8-a3a4-b84f3a898e0e-kube-api-access-wv6bg" (OuterVolumeSpecName: "kube-api-access-wv6bg") pod "808d5fda-088d-45f8-a3a4-b84f3a898e0e" (UID: "808d5fda-088d-45f8-a3a4-b84f3a898e0e"). InnerVolumeSpecName "kube-api-access-wv6bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:13:52.209200 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.209160 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808d5fda-088d-45f8-a3a4-b84f3a898e0e-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:13:52.209200 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.209192 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808d5fda-088d-45f8-a3a4-b84f3a898e0e-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:13:52.209200 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.209205 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6bg\" (UniqueName: \"kubernetes.io/projected/808d5fda-088d-45f8-a3a4-b84f3a898e0e-kube-api-access-wv6bg\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:13:52.223479 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.223450 2553 generic.go:356] "Generic (PLEG): container finished" podID="808d5fda-088d-45f8-a3a4-b84f3a898e0e" containerID="fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9" exitCode=0 Jul 14 20:13:52.223628 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.223494 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" Jul 14 20:13:52.223628 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.223536 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" event={"ID":"808d5fda-088d-45f8-a3a4-b84f3a898e0e","Type":"ContainerDied","Data":"fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9"} Jul 14 20:13:52.223628 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.223592 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp" event={"ID":"808d5fda-088d-45f8-a3a4-b84f3a898e0e","Type":"ContainerDied","Data":"01f19a28e25fafaf4e7ba3463823de015c68ab83e288f789d1465aea3acd3ecf"} Jul 14 20:13:52.223628 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.223616 2553 scope.go:117] "RemoveContainer" containerID="fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9" Jul 14 20:13:52.232751 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.232735 2553 scope.go:117] "RemoveContainer" containerID="fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9" Jul 14 20:13:52.233057 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:13:52.233032 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9\": container with ID starting with fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9 not found: ID does not exist" containerID="fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9" Jul 14 20:13:52.233127 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.233087 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9"} err="failed to get container status \"fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9\": rpc error: code = NotFound desc = could not find container \"fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9\": container with ID starting with fac956c6ae1cbd8bea015451b58afe279dbc6a5c80363d14eeb2fb853e6792a9 not found: ID does not exist" Jul 14 20:13:52.247999 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.247964 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp"] Jul 14 20:13:52.252397 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:52.252372 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f4a7e811-69ccc79444-cbmcp"] Jul 14 20:13:53.036459 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:53.036424 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808d5fda-088d-45f8-a3a4-b84f3a898e0e" path="/var/lib/kubelet/pods/808d5fda-088d-45f8-a3a4-b84f3a898e0e/volumes" Jul 14 20:13:53.138832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:53.138802 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:57.881832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:57.881800 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:57.982201 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:57.982163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:58.080510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080474 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh"] Jul 14 20:13:58.080863 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080851 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:58.080863 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080864 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080883 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4feb0822-1227-4a77-b917-bc6d057e5909" containerName="revoke-keys" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080888 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4feb0822-1227-4a77-b917-bc6d057e5909" containerName="revoke-keys" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080896 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808d5fda-088d-45f8-a3a4-b84f3a898e0e" containerName="maas-api" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080902 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="808d5fda-088d-45f8-a3a4-b84f3a898e0e" containerName="maas-api" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080950 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="808d5fda-088d-45f8-a3a4-b84f3a898e0e" containerName="maas-api" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080959 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaba4270-934f-41c0-92f3-41aa3b900240" containerName="revoke-keys" Jul 14 20:13:58.080992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.080967 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4feb0822-1227-4a77-b917-bc6d057e5909" containerName="revoke-keys" Jul 14 20:13:58.085272 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.085250 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.087818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.087787 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f61d89f4-openshift-default-dockercfg-jcws4\"" Jul 14 20:13:58.095628 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.095589 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh"] Jul 14 20:13:58.110250 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.110208 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:58.171268 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171164 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-socket\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171268 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171230 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hbf\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-kube-api-access-w6hbf\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171314 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-podinfo\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171346 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-credential-socket\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171364 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe4637-f793-4311-a8f3-225b356b8eb6-istiod-ca-cert\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171385 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-token\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171430 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-envoy\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171468 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-data\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.171700 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.171486 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-certs\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.202435 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.202386 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:58.272230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-data\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272233 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-certs\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272281 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-socket\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272319 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hbf\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-kube-api-access-w6hbf\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272350 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-podinfo\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272375 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-credential-socket\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272391 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe4637-f793-4311-a8f3-225b356b8eb6-istiod-ca-cert\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272413 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-token\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272510 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272444 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-envoy\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272828 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272626 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-data\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272828 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272689 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-certs\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272828 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272744 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-socket\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.272828 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.272773 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-credential-socket\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.273356 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.273338 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe4637-f793-4311-a8f3-225b356b8eb6-istiod-ca-cert\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.274721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.274698 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-envoy\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.274925 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.274906 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-podinfo\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.281749 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.281719 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-token\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.281915 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.281841 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hbf\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-kube-api-access-w6hbf\") pod \"e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.397114 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.397046 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:58.537932 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.537887 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh"] Jul 14 20:13:58.540129 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:13:58.540100 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fe4637_f793_4311_a8f3_225b356b8eb6.slice/crio-c422243c82b1b23dae252ed7eafe4d64038dcaed7f4ab21f8b97de1667103ff8 WatchSource:0}: Error finding container c422243c82b1b23dae252ed7eafe4d64038dcaed7f4ab21f8b97de1667103ff8: Status 404 returned error can't find the container with id c422243c82b1b23dae252ed7eafe4d64038dcaed7f4ab21f8b97de1667103ff8 Jul 14 20:13:58.542259 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.542223 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:13:58.542355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.542301 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:13:58.542355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:58.542339 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:13:59.251361 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.251333 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:59.262647 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.262616 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" event={"ID":"f9fe4637-f793-4311-a8f3-225b356b8eb6","Type":"ContainerStarted","Data":"61e385f4c726066787cbe5f368b805f0c0254fc5b5a1b48076cbcca586db3702"} Jul 14 20:13:59.262809 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.262672 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" event={"ID":"f9fe4637-f793-4311-a8f3-225b356b8eb6","Type":"ContainerStarted","Data":"c422243c82b1b23dae252ed7eafe4d64038dcaed7f4ab21f8b97de1667103ff8"} Jul 14 20:13:59.286447 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.286394 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podStartSLOduration=1.286375893 podStartE2EDuration="1.286375893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:13:59.284934873 +0000 UTC m=+1644.733561339" watchObservedRunningTime="2026-07-14 20:13:59.286375893 +0000 UTC m=+1644.735002339" Jul 14 20:13:59.322135 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.322098 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk"] Jul 14 20:13:59.325936 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.325906 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.329315 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.329280 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f61d89f4-serving-cert\"" Jul 14 20:13:59.338358 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.338333 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk"] Jul 14 20:13:59.345280 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.345242 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:59.398007 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.397961 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:59.403115 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.403054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:13:59.431955 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.431921 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:13:59.484968 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.484926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hqx\" (UniqueName: \"kubernetes.io/projected/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-kube-api-access-h6hqx\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.485157 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.484988 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-openshift-service-ca\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.485157 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.485037 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.585925 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.585814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hqx\" (UniqueName: \"kubernetes.io/projected/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-kube-api-access-h6hqx\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.585925 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.585899 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-openshift-service-ca\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.586181 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.585944 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.586181 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:13:59.586147 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-serving-cert: secret "maas-api-e2e-ait-f61d89f4-serving-cert" not found Jul 14 20:13:59.586267 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:13:59.586210 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls podName:3bf6eb02-1b1a-41f8-a8f2-4316c5752289 nodeName:}" failed. No retries permitted until 2026-07-14 20:14:00.086193308 +0000 UTC m=+1645.534819731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls") pod "maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" (UID: "3bf6eb02-1b1a-41f8-a8f2-4316c5752289") : secret "maas-api-e2e-ait-f61d89f4-serving-cert" not found Jul 14 20:13:59.586605 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.586585 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-openshift-service-ca\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.595777 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.595754 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hqx\" (UniqueName: \"kubernetes.io/projected/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-kube-api-access-h6hqx\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:13:59.960922 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:13:59.960883 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:14:00.061375 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.061343 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:14:00.092143 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.092106 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:14:00.094560 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.094536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls\") pod \"maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:14:00.238783 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.238690 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:14:00.269367 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.269328 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:14:00.270727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.270700 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:14:00.393677 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:00.393642 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk"] Jul 14 20:14:00.394536 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:14:00.394502 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf6eb02_1b1a_41f8_a8f2_4316c5752289.slice/crio-0f90083ea0a8075f8a2be174ce562806a994af759a8f6b1c1c65eb50325b0960 WatchSource:0}: Error finding container 0f90083ea0a8075f8a2be174ce562806a994af759a8f6b1c1c65eb50325b0960: Status 404 returned error can't find the container with id 0f90083ea0a8075f8a2be174ce562806a994af759a8f6b1c1c65eb50325b0960 Jul 14 20:14:01.181027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:01.180999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:14:01.275772 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:01.275730 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" event={"ID":"3bf6eb02-1b1a-41f8-a8f2-4316c5752289","Type":"ContainerStarted","Data":"c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1"} Jul 14 20:14:01.276300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:01.275794 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" event={"ID":"3bf6eb02-1b1a-41f8-a8f2-4316c5752289","Type":"ContainerStarted","Data":"0f90083ea0a8075f8a2be174ce562806a994af759a8f6b1c1c65eb50325b0960"} Jul 14 20:14:01.292287 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:01.292222 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" podStartSLOduration=1.79896283 podStartE2EDuration="2.292198156s" totalImagesPullingTime="493.235326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:14:01.291932503 +0000 UTC m=+1646.740558945" watchObservedRunningTime="2026-07-14 20:14:01.292198156 +0000 UTC m=+1646.740824627" Jul 14 20:14:02.281683 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:02.281642 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:14:08.294642 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:08.294614 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:14:12.337114 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:12.337082 2553 generic.go:356] "Generic (PLEG): container finished" podID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerID="61e385f4c726066787cbe5f368b805f0c0254fc5b5a1b48076cbcca586db3702" exitCode=0 Jul 14 20:14:12.337532 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:12.337115 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" event={"ID":"f9fe4637-f793-4311-a8f3-225b356b8eb6","Type":"ContainerDied","Data":"61e385f4c726066787cbe5f368b805f0c0254fc5b5a1b48076cbcca586db3702"} Jul 14 20:14:12.337575 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:12.337560 2553 scope.go:117] "RemoveContainer" containerID="61e385f4c726066787cbe5f368b805f0c0254fc5b5a1b48076cbcca586db3702" Jul 14 20:14:12.338622 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:12.338595 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:14:12.338689 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:12.338663 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:14:12.338726 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:12.338697 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:14:13.344146 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:13.344109 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" event={"ID":"f9fe4637-f793-4311-a8f3-225b356b8eb6","Type":"ContainerStarted","Data":"42c7fbf26fd1a504a3c68417d9714becc827710c7fbd1f503daf5af5a5cac314"} Jul 14 20:14:13.397238 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:13.397196 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:14:13.398525 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:13.398501 2553 patch_prober.go:28] interesting pod/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:14:13.398587 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:13.398553 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:14:14.398437 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:14.398398 2553 patch_prober.go:28] interesting pod/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:14:14.398842 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:14.398454 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:14:15.269232 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:15.269179 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:14:15.398294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:15.398254 2553 patch_prober.go:28] interesting pod/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:14:15.398483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:15.398317 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:14:16.397890 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:16.397855 2553 patch_prober.go:28] interesting pod/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:14:16.398086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:16.397911 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:14:17.397679 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:17.397642 2553 patch_prober.go:28] interesting pod/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:14:17.398081 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:17.397710 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:14:18.402137 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:18.402102 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:14:18.403166 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:14:18.403146 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:15:02.412415 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:02.412308 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:02.512426 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:02.512395 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:05.235804 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:05.235766 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:05.340591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:05.340560 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:15.038452 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.038424 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk"] Jul 14 20:15:15.038913 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.038611 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" podUID="3bf6eb02-1b1a-41f8-a8f2-4316c5752289" containerName="maas-api" containerID="cri-o://c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1" gracePeriod=30 Jul 14 20:15:15.067158 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.067116 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:15.283833 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.283805 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:15:15.380180 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.380147 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls\") pod \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " Jul 14 20:15:15.380337 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.380233 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-openshift-service-ca\") pod \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " Jul 14 20:15:15.380380 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.380361 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6hqx\" (UniqueName: \"kubernetes.io/projected/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-kube-api-access-h6hqx\") pod \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\" (UID: \"3bf6eb02-1b1a-41f8-a8f2-4316c5752289\") " Jul 14 20:15:15.380561 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.380537 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bf6eb02-1b1a-41f8-a8f2-4316c5752289" (UID: "3bf6eb02-1b1a-41f8-a8f2-4316c5752289"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:15:15.380650 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.380605 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:15.382346 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.382325 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bf6eb02-1b1a-41f8-a8f2-4316c5752289" (UID: "3bf6eb02-1b1a-41f8-a8f2-4316c5752289"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:15:15.382418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.382382 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-kube-api-access-h6hqx" (OuterVolumeSpecName: "kube-api-access-h6hqx") pod "3bf6eb02-1b1a-41f8-a8f2-4316c5752289" (UID: "3bf6eb02-1b1a-41f8-a8f2-4316c5752289"). InnerVolumeSpecName "kube-api-access-h6hqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:15:15.481031 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.480991 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6hqx\" (UniqueName: \"kubernetes.io/projected/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-kube-api-access-h6hqx\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:15.481031 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.481029 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6eb02-1b1a-41f8-a8f2-4316c5752289-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:15.678112 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.678006 2553 generic.go:356] "Generic (PLEG): container finished" podID="3bf6eb02-1b1a-41f8-a8f2-4316c5752289" containerID="c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1" exitCode=0 Jul 14 20:15:15.678112 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.678050 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" Jul 14 20:15:15.678112 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.678098 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" event={"ID":"3bf6eb02-1b1a-41f8-a8f2-4316c5752289","Type":"ContainerDied","Data":"c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1"} Jul 14 20:15:15.678323 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.678146 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk" event={"ID":"3bf6eb02-1b1a-41f8-a8f2-4316c5752289","Type":"ContainerDied","Data":"0f90083ea0a8075f8a2be174ce562806a994af759a8f6b1c1c65eb50325b0960"} Jul 14 20:15:15.678323 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.678168 2553 scope.go:117] "RemoveContainer" containerID="c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1" Jul 14 20:15:15.687435 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.687419 2553 scope.go:117] "RemoveContainer" containerID="c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1" Jul 14 20:15:15.687708 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:15:15.687692 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1\": container with ID starting with c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1 not found: ID does not exist" containerID="c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1" Jul 14 20:15:15.687752 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.687715 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1"} err="failed to get container status \"c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1\": rpc error: code = NotFound desc = could not find container \"c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1\": container with ID starting with c656cd43e346c7b006bb2abd31cf771bb826b8390b47442f2550e36bc4ec5ba1 not found: ID does not exist" Jul 14 20:15:15.704571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.704537 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk"] Jul 14 20:15:15.712559 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:15.712533 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f61d89f4-c558c4994-vbfjk"] Jul 14 20:15:17.036878 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:17.036844 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf6eb02-1b1a-41f8-a8f2-4316c5752289" path="/var/lib/kubelet/pods/3bf6eb02-1b1a-41f8-a8f2-4316c5752289/volumes" Jul 14 20:15:19.581966 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:19.581937 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:25.560591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:25.560558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:25.621810 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:25.621768 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:30.597372 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.597332 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh"] Jul 14 20:15:30.597763 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.597541 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" containerID="cri-o://42c7fbf26fd1a504a3c68417d9714becc827710c7fbd1f503daf5af5a5cac314" gracePeriod=30 Jul 14 20:15:30.619679 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.619622 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:30.884652 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.884619 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8"] Jul 14 20:15:30.885136 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.885115 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf6eb02-1b1a-41f8-a8f2-4316c5752289" containerName="maas-api" Jul 14 20:15:30.885136 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.885133 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf6eb02-1b1a-41f8-a8f2-4316c5752289" containerName="maas-api" Jul 14 20:15:30.885301 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.885222 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf6eb02-1b1a-41f8-a8f2-4316c5752289" containerName="maas-api" Jul 14 20:15:30.889653 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.889631 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:30.892002 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.891982 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e772d17e-openshift-default-dockercfg-wv7dz\"" Jul 14 20:15:30.902654 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.902630 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8"] Jul 14 20:15:30.918982 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:30.918953 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:31.026635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026597 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-certs\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026640 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-envoy\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026733 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a9074-747e-4ef1-b308-0e53f267ecff-istiod-ca-cert\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026774 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znss\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-kube-api-access-6znss\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026799 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-token\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026980 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026830 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-credential-socket\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026980 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026866 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-socket\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026980 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026891 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-data\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.026980 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.026973 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-podinfo\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128455 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128408 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-podinfo\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128748 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128627 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-certs\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128748 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128691 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-envoy\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128748 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128742 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a9074-747e-4ef1-b308-0e53f267ecff-istiod-ca-cert\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128946 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128761 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znss\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-kube-api-access-6znss\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128946 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128800 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-token\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128946 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-credential-socket\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128946 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128865 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-socket\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.128946 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.128888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-data\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.129259 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.129013 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-certs\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.129259 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.129208 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-socket\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.129374 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.129343 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-credential-socket\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.129425 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.129392 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-data\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.129505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.129455 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a9074-747e-4ef1-b308-0e53f267ecff-istiod-ca-cert\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.130973 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.130953 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-envoy\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.131085 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.130998 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-podinfo\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.138434 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.138377 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-token\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.138546 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.138531 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znss\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-kube-api-access-6znss\") pod \"e2e-derive-e772d17e-openshift-default-847f94c555-27rb8\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.200480 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.200445 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:31.344394 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.344366 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8"] Jul 14 20:15:31.345690 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:15:31.345660 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a9074_747e_4ef1_b308_0e53f267ecff.slice/crio-44225e7ce762dd049c0baa4ccc006bdb59e53daf270b22741b4978c0ce447a7a WatchSource:0}: Error finding container 44225e7ce762dd049c0baa4ccc006bdb59e53daf270b22741b4978c0ce447a7a: Status 404 returned error can't find the container with id 44225e7ce762dd049c0baa4ccc006bdb59e53daf270b22741b4978c0ce447a7a Jul 14 20:15:31.347849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.347818 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:15:31.347915 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.347883 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:15:31.347952 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.347918 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:15:31.771258 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.771164 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" event={"ID":"2a0a9074-747e-4ef1-b308-0e53f267ecff","Type":"ContainerStarted","Data":"5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287"} Jul 14 20:15:31.771258 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.771218 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" event={"ID":"2a0a9074-747e-4ef1-b308-0e53f267ecff","Type":"ContainerStarted","Data":"44225e7ce762dd049c0baa4ccc006bdb59e53daf270b22741b4978c0ce447a7a"} Jul 14 20:15:31.791696 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:31.791641 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" podStartSLOduration=1.791625928 podStartE2EDuration="1.791625928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:15:31.790307901 +0000 UTC m=+1737.238934349" watchObservedRunningTime="2026-07-14 20:15:31.791625928 +0000 UTC m=+1737.240252374" Jul 14 20:15:32.026672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.026575 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:32.143540 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.143494 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b"] Jul 14 20:15:32.144202 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.144172 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:32.147155 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.147134 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.150049 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.150024 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e772d17e-serving-cert\"" Jul 14 20:15:32.168031 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.167994 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b"] Jul 14 20:15:32.201252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.201217 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:32.206292 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.206268 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:32.240159 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.240121 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sg5p\" (UniqueName: \"kubernetes.io/projected/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-kube-api-access-2sg5p\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.240360 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.240188 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.240360 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.240275 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-openshift-service-ca\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.341173 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.341085 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.341173 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.341153 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-openshift-service-ca\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.341377 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:15:32.341192 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-serving-cert: secret "maas-api-e2e-derive-e772d17e-serving-cert" not found Jul 14 20:15:32.341377 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:15:32.341264 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls podName:b8ce4474-4e10-4c2c-a9bd-f513b2a881ae nodeName:}" failed. No retries permitted until 2026-07-14 20:15:32.841243412 +0000 UTC m=+1738.289869852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls") pod "maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" (UID: "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae") : secret "maas-api-e2e-derive-e772d17e-serving-cert" not found Jul 14 20:15:32.341377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.341332 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sg5p\" (UniqueName: \"kubernetes.io/projected/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-kube-api-access-2sg5p\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.341787 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.341769 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-openshift-service-ca\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.359977 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.359944 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sg5p\" (UniqueName: \"kubernetes.io/projected/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-kube-api-access-2sg5p\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.777558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.777529 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:32.778649 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.778632 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:32.846753 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.846721 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.849202 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.849172 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls\") pod \"maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:32.924844 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:32.924795 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:33.024591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:33.024558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:33.058533 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:33.058435 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:33.198584 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:33.198560 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b"] Jul 14 20:15:33.199734 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:15:33.199707 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ce4474_4e10_4c2c_a9bd_f513b2a881ae.slice/crio-3436f750276de246218a227ba47334e6e91f2f4309a4638685ae0d97199bacd6 WatchSource:0}: Error finding container 3436f750276de246218a227ba47334e6e91f2f4309a4638685ae0d97199bacd6: Status 404 returned error can't find the container with id 3436f750276de246218a227ba47334e6e91f2f4309a4638685ae0d97199bacd6 Jul 14 20:15:33.404913 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:33.404873 2553 patch_prober.go:28] interesting pod/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:15:33.405134 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:33.404937 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:15:33.783453 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:33.783425 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" event={"ID":"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae","Type":"ContainerStarted","Data":"3436f750276de246218a227ba47334e6e91f2f4309a4638685ae0d97199bacd6"} Jul 14 20:15:34.796731 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:34.796696 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" event={"ID":"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae","Type":"ContainerStarted","Data":"8fc0704bcfd3556d23a9003656a788c038eac4164e2f5883fdeec24111e74527"} Jul 14 20:15:34.797190 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:34.796969 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:34.817111 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:34.817026 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" podStartSLOduration=2.261199349 podStartE2EDuration="2.817010894s" totalImagesPullingTime="555.811545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:15:34.815659883 +0000 UTC m=+1740.264286352" watchObservedRunningTime="2026-07-14 20:15:34.817010894 +0000 UTC m=+1740.265637339" Jul 14 20:15:35.804591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.804539 2553 generic.go:356] "Generic (PLEG): container finished" podID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerID="42c7fbf26fd1a504a3c68417d9714becc827710c7fbd1f503daf5af5a5cac314" exitCode=0 Jul 14 20:15:35.805043 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.804931 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" event={"ID":"f9fe4637-f793-4311-a8f3-225b356b8eb6","Type":"ContainerDied","Data":"42c7fbf26fd1a504a3c68417d9714becc827710c7fbd1f503daf5af5a5cac314"} Jul 14 20:15:35.805043 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.805005 2553 scope.go:117] "RemoveContainer" containerID="61e385f4c726066787cbe5f368b805f0c0254fc5b5a1b48076cbcca586db3702" Jul 14 20:15:35.898023 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.897992 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.986445 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6hbf\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-kube-api-access-w6hbf\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987012 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-envoy\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987073 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-certs\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987176 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-podinfo\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987227 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-token\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987257 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-credential-socket\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987300 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe4637-f793-4311-a8f3-225b356b8eb6-istiod-ca-cert\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987382 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-data\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.987419 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-socket\") pod \"f9fe4637-f793-4311-a8f3-225b356b8eb6\" (UID: \"f9fe4637-f793-4311-a8f3-225b356b8eb6\") " Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.988346 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.989618 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.989638 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fe4637-f793-4311-a8f3-225b356b8eb6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.989886 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.990482 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe4637-f793-4311-a8f3-225b356b8eb6-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.990502 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.990517 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:35.991086 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.990531 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:35.994560 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.994514 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:35.995223 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.995195 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:15:35.995327 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.995281 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:35.996110 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.996047 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:15:35.999534 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:35.999471 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-kube-api-access-w6hbf" (OuterVolumeSpecName: "kube-api-access-w6hbf") pod "f9fe4637-f793-4311-a8f3-225b356b8eb6" (UID: "f9fe4637-f793-4311-a8f3-225b356b8eb6"). InnerVolumeSpecName "kube-api-access-w6hbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:15:36.091691 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.091638 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hbf\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-kube-api-access-w6hbf\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:36.091691 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.091686 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:36.091691 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.091696 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:36.091691 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.091704 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fe4637-f793-4311-a8f3-225b356b8eb6-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:36.092023 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.091712 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fe4637-f793-4311-a8f3-225b356b8eb6-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:15:36.811325 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.811283 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" event={"ID":"f9fe4637-f793-4311-a8f3-225b356b8eb6","Type":"ContainerDied","Data":"c422243c82b1b23dae252ed7eafe4d64038dcaed7f4ab21f8b97de1667103ff8"} Jul 14 20:15:36.811761 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.811343 2553 scope.go:117] "RemoveContainer" containerID="42c7fbf26fd1a504a3c68417d9714becc827710c7fbd1f503daf5af5a5cac314" Jul 14 20:15:36.811761 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.811301 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh" Jul 14 20:15:36.834840 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.834811 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh"] Jul 14 20:15:36.838591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.838564 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f61d89f4-openshift-default-7d46b97b7f-xm5qh"] Jul 14 20:15:36.943514 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.943481 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:36.998644 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:36.998608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:37.036809 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:37.036775 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" path="/var/lib/kubelet/pods/f9fe4637-f793-4311-a8f3-225b356b8eb6/volumes" Jul 14 20:15:40.218415 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:40.218376 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:15:40.812162 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:40.812130 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:15:42.848681 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:42.848651 2553 generic.go:356] "Generic (PLEG): container finished" podID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerID="5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287" exitCode=0 Jul 14 20:15:42.849126 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:42.848732 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" event={"ID":"2a0a9074-747e-4ef1-b308-0e53f267ecff","Type":"ContainerDied","Data":"5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287"} Jul 14 20:15:42.849175 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:42.849136 2553 scope.go:117] "RemoveContainer" containerID="5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287" Jul 14 20:15:42.850089 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:42.850049 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:15:42.850157 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:42.850132 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:15:42.850194 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:42.850163 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:15:43.856777 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:43.856738 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" event={"ID":"2a0a9074-747e-4ef1-b308-0e53f267ecff","Type":"ContainerStarted","Data":"223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19"} Jul 14 20:15:44.201275 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:44.201230 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:44.202538 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:44.202508 2553 patch_prober.go:28] interesting pod/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 14 20:15:44.202672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:44.202575 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 14 20:15:45.201373 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:45.201323 2553 patch_prober.go:28] interesting pod/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 14 20:15:45.201867 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:45.201411 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 14 20:15:46.201821 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:46.201782 2553 patch_prober.go:28] interesting pod/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 14 20:15:46.202270 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:46.201839 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 14 20:15:47.201454 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:47.201418 2553 patch_prober.go:28] interesting pod/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 14 20:15:47.201608 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:47.201484 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 14 20:15:47.777733 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:47.777653 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:48.205096 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:48.205049 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:15:48.206057 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:15:48.206035 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:16:35.067592 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:35.067562 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:16:35.072125 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:35.072102 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:16:36.994491 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:36.994455 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b"] Jul 14 20:16:36.994974 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:36.994751 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" podUID="b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" containerName="maas-api" containerID="cri-o://8fc0704bcfd3556d23a9003656a788c038eac4164e2f5883fdeec24111e74527" gracePeriod=30 Jul 14 20:16:37.142350 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.142302 2553 generic.go:356] "Generic (PLEG): container finished" podID="b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" containerID="8fc0704bcfd3556d23a9003656a788c038eac4164e2f5883fdeec24111e74527" exitCode=0 Jul 14 20:16:37.142507 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.142369 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" event={"ID":"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae","Type":"ContainerDied","Data":"8fc0704bcfd3556d23a9003656a788c038eac4164e2f5883fdeec24111e74527"} Jul 14 20:16:37.245427 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.245363 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:16:37.349800 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.349761 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sg5p\" (UniqueName: \"kubernetes.io/projected/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-kube-api-access-2sg5p\") pod \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " Jul 14 20:16:37.349970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.349852 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls\") pod \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " Jul 14 20:16:37.349970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.349889 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-openshift-service-ca\") pod \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\" (UID: \"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae\") " Jul 14 20:16:37.350337 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.350306 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" (UID: "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:16:37.352020 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.351993 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" (UID: "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:16:37.352020 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.352001 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-kube-api-access-2sg5p" (OuterVolumeSpecName: "kube-api-access-2sg5p") pod "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" (UID: "b8ce4474-4e10-4c2c-a9bd-f513b2a881ae"). InnerVolumeSpecName "kube-api-access-2sg5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:16:37.450641 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.450592 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:37.450641 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.450637 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:37.450641 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.450647 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sg5p\" (UniqueName: \"kubernetes.io/projected/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae-kube-api-access-2sg5p\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:37.687243 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:37.687211 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:38.150566 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:38.150538 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" Jul 14 20:16:38.151024 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:38.150546 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b" event={"ID":"b8ce4474-4e10-4c2c-a9bd-f513b2a881ae","Type":"ContainerDied","Data":"3436f750276de246218a227ba47334e6e91f2f4309a4638685ae0d97199bacd6"} Jul 14 20:16:38.151024 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:38.150670 2553 scope.go:117] "RemoveContainer" containerID="8fc0704bcfd3556d23a9003656a788c038eac4164e2f5883fdeec24111e74527" Jul 14 20:16:38.173875 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:38.173847 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b"] Jul 14 20:16:38.178125 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:38.178102 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e772d17e-68fffcf6f-dgm9b"] Jul 14 20:16:39.036812 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:39.036777 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" path="/var/lib/kubelet/pods/b8ce4474-4e10-4c2c-a9bd-f513b2a881ae/volumes" Jul 14 20:16:41.626138 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:41.626091 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:47.687324 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:47.687291 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:47.762632 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:47.762582 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:51.814514 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:51.814470 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8"] Jul 14 20:16:51.815036 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:51.814688 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" containerID="cri-o://223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19" gracePeriod=30 Jul 14 20:16:51.854865 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:51.854832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:52.513236 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:52.513198 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:52.606496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:52.606450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:52.727003 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:52.726972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:54.334202 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:54.334154 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:16:57.053351 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.053325 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:16:57.133478 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133440 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a9074-747e-4ef1-b308-0e53f267ecff-istiod-ca-cert\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133501 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-token\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133523 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-credential-socket\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133579 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-socket\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133611 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6znss\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-kube-api-access-6znss\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133644 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-data\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133686 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-envoy\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.133727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133717 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-certs\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.134144 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133745 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-podinfo\") pod \"2a0a9074-747e-4ef1-b308-0e53f267ecff\" (UID: \"2a0a9074-747e-4ef1-b308-0e53f267ecff\") " Jul 14 20:16:57.134144 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.133764 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0a9074-747e-4ef1-b308-0e53f267ecff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:16:57.134144 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.134015 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a9074-747e-4ef1-b308-0e53f267ecff-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.134313 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.134220 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:57.134477 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.134443 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:57.134685 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.134645 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:57.134685 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.134638 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:57.136183 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.136158 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-kube-api-access-6znss" (OuterVolumeSpecName: "kube-api-access-6znss") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "kube-api-access-6znss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:16:57.136286 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.136187 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:16:57.136471 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.136451 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:16:57.136583 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.136531 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a0a9074-747e-4ef1-b308-0e53f267ecff" (UID: "2a0a9074-747e-4ef1-b308-0e53f267ecff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:57.234818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234784 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.234818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234812 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.234818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234820 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.235038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234828 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.235038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234838 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.235038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234848 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.235038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234857 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znss\" (UniqueName: \"kubernetes.io/projected/2a0a9074-747e-4ef1-b308-0e53f267ecff-kube-api-access-6znss\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.235038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.234865 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a9074-747e-4ef1-b308-0e53f267ecff-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:16:57.253472 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.253439 2553 generic.go:356] "Generic (PLEG): container finished" podID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerID="223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19" exitCode=0 Jul 14 20:16:57.253611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.253480 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" Jul 14 20:16:57.253611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.253531 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" event={"ID":"2a0a9074-747e-4ef1-b308-0e53f267ecff","Type":"ContainerDied","Data":"223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19"} Jul 14 20:16:57.253611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.253584 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8" event={"ID":"2a0a9074-747e-4ef1-b308-0e53f267ecff","Type":"ContainerDied","Data":"44225e7ce762dd049c0baa4ccc006bdb59e53daf270b22741b4978c0ce447a7a"} Jul 14 20:16:57.253611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.253606 2553 scope.go:117] "RemoveContainer" containerID="223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19" Jul 14 20:16:57.263055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.263037 2553 scope.go:117] "RemoveContainer" containerID="5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287" Jul 14 20:16:57.270870 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.270854 2553 scope.go:117] "RemoveContainer" containerID="223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19" Jul 14 20:16:57.271124 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:16:57.271105 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19\": container with ID starting with 223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19 not found: ID does not exist" containerID="223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19" Jul 14 20:16:57.271201 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.271136 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19"} err="failed to get container status \"223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19\": rpc error: code = NotFound desc = could not find container \"223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19\": container with ID starting with 223249a7155c2a546974f5c175941bc3e0b567164bbcfef51de1f7ad55c70c19 not found: ID does not exist" Jul 14 20:16:57.271201 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.271160 2553 scope.go:117] "RemoveContainer" containerID="5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287" Jul 14 20:16:57.271392 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:16:57.271375 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287\": container with ID starting with 5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287 not found: ID does not exist" containerID="5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287" Jul 14 20:16:57.271434 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.271398 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287"} err="failed to get container status \"5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287\": rpc error: code = NotFound desc = could not find container \"5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287\": container with ID starting with 5fabc17d34267265663a4da1d12f582c9f1e420bcfcbb5f7f9df92e5156e5287 not found: ID does not exist" Jul 14 20:16:57.276949 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.276927 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8"] Jul 14 20:16:57.281750 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:57.281731 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e772d17e-openshift-default-847f94c555-27rb8"] Jul 14 20:16:59.036484 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:16:59.036454 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" path="/var/lib/kubelet/pods/2a0a9074-747e-4ef1-b308-0e53f267ecff/volumes" Jul 14 20:17:00.484852 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:00.484821 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:00.558922 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:00.558887 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:00.638402 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:00.638371 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:00.736241 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:00.736163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:00.835942 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:00.835912 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:01.251841 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:01.251806 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:03.445667 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:03.445635 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:27.354842 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:27.354809 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:33.345922 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:33.345879 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:33.406657 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:33.406621 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:33.560771 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:33.560720 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:33.706776 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:33.706739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:49.584123 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:49.584085 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:55.612685 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:55.612651 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:55.716097 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:55.716054 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:17:55.817533 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:17:55.817498 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:06.399862 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:06.399784 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:07.296555 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:07.296516 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:11.229840 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:11.229803 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:13.108257 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:13.108226 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:13.212352 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:13.212324 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:13.307322 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:13.307277 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:13.406698 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:13.406663 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:14.259956 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:14.259913 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:14.354120 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:14.354047 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:14.466165 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:14.466137 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:16.188131 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:16.188100 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:22.273204 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:22.273174 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:22.370232 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:22.370194 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:22.961915 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:22.961877 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:29.074468 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.074436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:29.174717 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.174687 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:29.343288 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.343204 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:29.629337 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629302 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l"] Jul 14 20:18:29.629643 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629630 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" containerName="maas-api" Jul 14 20:18:29.629692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629644 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" containerName="maas-api" Jul 14 20:18:29.629692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629659 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" Jul 14 20:18:29.629692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629664 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" Jul 14 20:18:29.629692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629672 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" Jul 14 20:18:29.629692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629679 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" Jul 14 20:18:29.629692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629689 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629696 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629703 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629708 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629765 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ce4474-4e10-4c2c-a9bd-f513b2a881ae" containerName="maas-api" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629772 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629778 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fe4637-f793-4311-a8f3-225b356b8eb6" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629784 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" Jul 14 20:18:29.629876 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.629791 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0a9074-747e-4ef1-b308-0e53f267ecff" containerName="istio-proxy" Jul 14 20:18:29.632903 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.632887 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.635765 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.635745 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ad8efdd2-openshift-default-dockercfg-9ghwl\"" Jul 14 20:18:29.663433 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.663400 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:29.664251 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.664229 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l"] Jul 14 20:18:29.749319 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749270 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-certs\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749331 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-socket\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749353 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-data\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749408 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52z5\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-kube-api-access-k52z5\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749434 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-credential-socket\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749473 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76db8bb-2de9-4279-b327-46c4122a4e74-istiod-ca-cert\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749713 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749579 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-podinfo\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749713 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749611 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-envoy\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.749713 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.749646 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-token\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.761597 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.761574 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:29.850624 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-certs\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.850795 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-socket\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.850795 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-data\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.850795 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52z5\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-kube-api-access-k52z5\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.850795 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850717 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-credential-socket\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.850795 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850763 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76db8bb-2de9-4279-b327-46c4122a4e74-istiod-ca-cert\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851051 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850803 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-podinfo\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851051 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850829 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-envoy\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851051 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.850863 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-token\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851051 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.851021 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-certs\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851318 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.851157 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-credential-socket\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851318 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.851215 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-socket\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851394 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.851358 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-data\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.851504 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.851486 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76db8bb-2de9-4279-b327-46c4122a4e74-istiod-ca-cert\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.853115 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.853095 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-envoy\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.853260 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.853242 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-podinfo\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.860227 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.860203 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-token\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.860396 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.860379 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52z5\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-kube-api-access-k52z5\") pod \"e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:29.942230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:29.942136 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:30.076167 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.076139 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l"] Jul 14 20:18:30.078086 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:18:30.078038 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76db8bb_2de9_4279_b327_46c4122a4e74.slice/crio-6b2c4e1649d3a9b02e740e9c00f1f33457b01f999aaa3f8810681cf8246b83cb WatchSource:0}: Error finding container 6b2c4e1649d3a9b02e740e9c00f1f33457b01f999aaa3f8810681cf8246b83cb: Status 404 returned error can't find the container with id 6b2c4e1649d3a9b02e740e9c00f1f33457b01f999aaa3f8810681cf8246b83cb Jul 14 20:18:30.080141 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.080108 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:18:30.080257 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.080170 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:18:30.080257 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.080198 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:18:30.728243 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.728207 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" event={"ID":"c76db8bb-2de9-4279-b327-46c4122a4e74","Type":"ContainerStarted","Data":"747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c"} Jul 14 20:18:30.728406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.728266 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" event={"ID":"c76db8bb-2de9-4279-b327-46c4122a4e74","Type":"ContainerStarted","Data":"6b2c4e1649d3a9b02e740e9c00f1f33457b01f999aaa3f8810681cf8246b83cb"} Jul 14 20:18:30.749344 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.749289 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" podStartSLOduration=1.7492739149999998 podStartE2EDuration="1.749273915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:18:30.748771578 +0000 UTC m=+1916.197398021" watchObservedRunningTime="2026-07-14 20:18:30.749273915 +0000 UTC m=+1916.197900358" Jul 14 20:18:30.942871 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:30.942831 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:31.947433 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:31.947400 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:32.673049 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.673018 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:32.738585 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.738552 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:32.739689 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.739660 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:32.760762 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.760734 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:32.776119 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.776085 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4"] Jul 14 20:18:32.779736 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.779718 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.782221 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.782198 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ad8efdd2-serving-cert\"" Jul 14 20:18:32.789638 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.789610 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4"] Jul 14 20:18:32.853240 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.853198 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:32.878495 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.878463 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s29dt\" (UniqueName: \"kubernetes.io/projected/08948fc9-44c6-4260-8b25-179dc20568e5-kube-api-access-s29dt\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.878650 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.878533 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08948fc9-44c6-4260-8b25-179dc20568e5-openshift-service-ca\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.878710 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.878686 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.979772 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.979673 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08948fc9-44c6-4260-8b25-179dc20568e5-openshift-service-ca\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.979772 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.979747 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.980339 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.979801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s29dt\" (UniqueName: \"kubernetes.io/projected/08948fc9-44c6-4260-8b25-179dc20568e5-kube-api-access-s29dt\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.980339 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:18:32.979850 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-serving-cert: secret "maas-api-e2e-mt-ad8efdd2-serving-cert" not found Jul 14 20:18:32.980339 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:18:32.979926 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls podName:08948fc9-44c6-4260-8b25-179dc20568e5 nodeName:}" failed. No retries permitted until 2026-07-14 20:18:33.479907036 +0000 UTC m=+1918.928533473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls") pod "maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" (UID: "08948fc9-44c6-4260-8b25-179dc20568e5") : secret "maas-api-e2e-mt-ad8efdd2-serving-cert" not found Jul 14 20:18:32.980649 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.980499 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08948fc9-44c6-4260-8b25-179dc20568e5-openshift-service-ca\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:32.999271 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:32.999238 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s29dt\" (UniqueName: \"kubernetes.io/projected/08948fc9-44c6-4260-8b25-179dc20568e5-kube-api-access-s29dt\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:33.111545 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.111513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:33.484643 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.484608 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:33.487205 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.487173 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls\") pod \"maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:33.499607 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.499578 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:33.695313 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.695268 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:33.833643 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.833621 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4"] Jul 14 20:18:33.835381 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:18:33.835354 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08948fc9_44c6_4260_8b25_179dc20568e5.slice/crio-3a879c59c5ffa888407c5a88d645d51a631801fa0c788232c6a5578379435ed8 WatchSource:0}: Error finding container 3a879c59c5ffa888407c5a88d645d51a631801fa0c788232c6a5578379435ed8: Status 404 returned error can't find the container with id 3a879c59c5ffa888407c5a88d645d51a631801fa0c788232c6a5578379435ed8 Jul 14 20:18:33.837197 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:33.837180 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:18:34.099456 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:34.099371 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:34.750208 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:34.750174 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" event={"ID":"08948fc9-44c6-4260-8b25-179dc20568e5","Type":"ContainerStarted","Data":"da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce"} Jul 14 20:18:34.750208 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:34.750227 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" event={"ID":"08948fc9-44c6-4260-8b25-179dc20568e5","Type":"ContainerStarted","Data":"3a879c59c5ffa888407c5a88d645d51a631801fa0c788232c6a5578379435ed8"} Jul 14 20:18:34.750489 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:34.750457 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:34.781245 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:34.781197 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" podStartSLOduration=2.368285103 podStartE2EDuration="2.78118019s" totalImagesPullingTime="412.895087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:18:34.779590239 +0000 UTC m=+1920.228216683" watchObservedRunningTime="2026-07-14 20:18:34.78118019 +0000 UTC m=+1920.229806670" Jul 14 20:18:37.216796 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:37.216746 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:37.929121 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:37.929086 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:38.331791 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:38.331694 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:38.432418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:38.432384 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:40.760981 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:40.760953 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:44.312105 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:44.312053 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:44.411885 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:44.411846 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:46.820300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:46.820271 2553 generic.go:356] "Generic (PLEG): container finished" podID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerID="747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c" exitCode=0 Jul 14 20:18:46.820702 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:46.820346 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" event={"ID":"c76db8bb-2de9-4279-b327-46c4122a4e74","Type":"ContainerDied","Data":"747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c"} Jul 14 20:18:46.820747 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:46.820723 2553 scope.go:117] "RemoveContainer" containerID="747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c" Jul 14 20:18:46.821662 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:46.821635 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:18:46.821719 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:46.821707 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:18:46.821754 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:46.821741 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:18:47.738591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:47.738546 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:47.827527 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:47.827489 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" event={"ID":"c76db8bb-2de9-4279-b327-46c4122a4e74","Type":"ContainerStarted","Data":"75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312"} Jul 14 20:18:47.942367 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:47.942325 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:47.943589 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:47.943565 2553 patch_prober.go:28] interesting pod/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:18:47.943744 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:47.943612 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:18:48.943000 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:48.942961 2553 patch_prober.go:28] interesting pod/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:18:48.943442 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:48.943014 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:18:49.942485 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:49.942450 2553 patch_prober.go:28] interesting pod/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:18:49.942659 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:49.942502 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:18:50.942632 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:50.942592 2553 patch_prober.go:28] interesting pod/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:18:50.943024 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:50.942658 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:18:51.947573 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:51.947537 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:51.947960 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:51.947806 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:51.948600 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:51.948581 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:18:55.261622 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.261582 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4"] Jul 14 20:18:55.262054 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.261858 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" podUID="08948fc9-44c6-4260-8b25-179dc20568e5" containerName="maas-api" containerID="cri-o://da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce" gracePeriod=30 Jul 14 20:18:55.297288 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.297253 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:18:55.512526 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.512460 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:55.587041 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.587002 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls\") pod \"08948fc9-44c6-4260-8b25-179dc20568e5\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " Jul 14 20:18:55.587250 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.587103 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08948fc9-44c6-4260-8b25-179dc20568e5-openshift-service-ca\") pod \"08948fc9-44c6-4260-8b25-179dc20568e5\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " Jul 14 20:18:55.587250 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.587201 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s29dt\" (UniqueName: \"kubernetes.io/projected/08948fc9-44c6-4260-8b25-179dc20568e5-kube-api-access-s29dt\") pod \"08948fc9-44c6-4260-8b25-179dc20568e5\" (UID: \"08948fc9-44c6-4260-8b25-179dc20568e5\") " Jul 14 20:18:55.587468 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.587445 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08948fc9-44c6-4260-8b25-179dc20568e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08948fc9-44c6-4260-8b25-179dc20568e5" (UID: "08948fc9-44c6-4260-8b25-179dc20568e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:18:55.589278 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.589253 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08948fc9-44c6-4260-8b25-179dc20568e5-kube-api-access-s29dt" (OuterVolumeSpecName: "kube-api-access-s29dt") pod "08948fc9-44c6-4260-8b25-179dc20568e5" (UID: "08948fc9-44c6-4260-8b25-179dc20568e5"). InnerVolumeSpecName "kube-api-access-s29dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:18:55.589356 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.589333 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08948fc9-44c6-4260-8b25-179dc20568e5" (UID: "08948fc9-44c6-4260-8b25-179dc20568e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:18:55.688152 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.688121 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08948fc9-44c6-4260-8b25-179dc20568e5-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:18:55.688152 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.688151 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08948fc9-44c6-4260-8b25-179dc20568e5-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:18:55.688152 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.688160 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s29dt\" (UniqueName: \"kubernetes.io/projected/08948fc9-44c6-4260-8b25-179dc20568e5-kube-api-access-s29dt\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:18:55.871843 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.871753 2553 generic.go:356] "Generic (PLEG): container finished" podID="08948fc9-44c6-4260-8b25-179dc20568e5" containerID="da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce" exitCode=0 Jul 14 20:18:55.871843 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.871793 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" Jul 14 20:18:55.871843 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.871799 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" event={"ID":"08948fc9-44c6-4260-8b25-179dc20568e5","Type":"ContainerDied","Data":"da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce"} Jul 14 20:18:55.871843 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.871844 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4" event={"ID":"08948fc9-44c6-4260-8b25-179dc20568e5","Type":"ContainerDied","Data":"3a879c59c5ffa888407c5a88d645d51a631801fa0c788232c6a5578379435ed8"} Jul 14 20:18:55.872178 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.871866 2553 scope.go:117] "RemoveContainer" containerID="da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce" Jul 14 20:18:55.881180 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.881160 2553 scope.go:117] "RemoveContainer" containerID="da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce" Jul 14 20:18:55.881418 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:18:55.881399 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce\": container with ID starting with da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce not found: ID does not exist" containerID="da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce" Jul 14 20:18:55.881473 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.881427 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce"} err="failed to get container status \"da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce\": rpc error: code = NotFound desc = could not find container \"da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce\": container with ID starting with da3d136fc5a20c323d276e48d07641f2af350f385cf9972f50a44a40729560ce not found: ID does not exist" Jul 14 20:18:55.894861 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.894836 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4"] Jul 14 20:18:55.900905 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:55.900881 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8efdd2-84d6c8f9dc-72zj4"] Jul 14 20:18:57.036242 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:57.036207 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08948fc9-44c6-4260-8b25-179dc20568e5" path="/var/lib/kubelet/pods/08948fc9-44c6-4260-8b25-179dc20568e5/volumes" Jul 14 20:18:59.778356 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:18:59.778304 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:06.333253 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:06.333212 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:06.432384 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:06.432324 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:10.488921 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:10.488883 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l"] Jul 14 20:19:10.489362 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:10.489136 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" containerID="cri-o://75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312" gracePeriod=30 Jul 14 20:19:10.512468 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:10.512431 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:10.734501 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:10.734462 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:10.901258 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:10.901199 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:11.019557 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:11.019514 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:11.112931 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:11.112892 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:15.739926 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.739901 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:19:15.881434 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881396 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-certs\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881434 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881436 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-data\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881462 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-podinfo\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881501 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-envoy\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881582 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k52z5\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-kube-api-access-k52z5\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881606 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-credential-socket\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881635 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-token\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.881692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881692 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-socket\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.882027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881720 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76db8bb-2de9-4279-b327-46c4122a4e74-istiod-ca-cert\") pod \"c76db8bb-2de9-4279-b327-46c4122a4e74\" (UID: \"c76db8bb-2de9-4279-b327-46c4122a4e74\") " Jul 14 20:19:15.882027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881745 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:19:15.882027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881966 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:19:15.882027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881992 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.882027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.881977 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:19:15.882336 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.882289 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76db8bb-2de9-4279-b327-46c4122a4e74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:19:15.882508 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.882478 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-data" (OuterVolumeSpecName: "istio-data") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:19:15.883818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.883794 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:19:15.883913 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.883853 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-kube-api-access-k52z5" (OuterVolumeSpecName: "kube-api-access-k52z5") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "kube-api-access-k52z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:19:15.883913 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.883866 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:19:15.883913 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.883892 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-token" (OuterVolumeSpecName: "istio-token") pod "c76db8bb-2de9-4279-b327-46c4122a4e74" (UID: "c76db8bb-2de9-4279-b327-46c4122a4e74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:19:15.980995 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.980955 2553 generic.go:356] "Generic (PLEG): container finished" podID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerID="75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312" exitCode=0 Jul 14 20:19:15.981188 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.981006 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" Jul 14 20:19:15.981188 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.981008 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" event={"ID":"c76db8bb-2de9-4279-b327-46c4122a4e74","Type":"ContainerDied","Data":"75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312"} Jul 14 20:19:15.981188 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.981097 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l" event={"ID":"c76db8bb-2de9-4279-b327-46c4122a4e74","Type":"ContainerDied","Data":"6b2c4e1649d3a9b02e740e9c00f1f33457b01f999aaa3f8810681cf8246b83cb"} Jul 14 20:19:15.981188 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.981136 2553 scope.go:117] "RemoveContainer" containerID="75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312" Jul 14 20:19:15.982412 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982392 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52z5\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-kube-api-access-k52z5\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982412 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982412 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982422 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982432 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982439 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76db8bb-2de9-4279-b327-46c4122a4e74-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982449 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982456 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.982558 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.982464 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76db8bb-2de9-4279-b327-46c4122a4e74-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:19:15.992968 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:15.992948 2553 scope.go:117] "RemoveContainer" containerID="747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c" Jul 14 20:19:16.001679 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.001660 2553 scope.go:117] "RemoveContainer" containerID="75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312" Jul 14 20:19:16.001917 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:19:16.001899 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312\": container with ID starting with 75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312 not found: ID does not exist" containerID="75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312" Jul 14 20:19:16.001979 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.001924 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312"} err="failed to get container status \"75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312\": rpc error: code = NotFound desc = could not find container \"75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312\": container with ID starting with 75545866086fede9cf0dc95764752d37bd7dfd82dec1f12fb0770e4227729312 not found: ID does not exist" Jul 14 20:19:16.001979 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.001941 2553 scope.go:117] "RemoveContainer" containerID="747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c" Jul 14 20:19:16.002178 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:19:16.002161 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c\": container with ID starting with 747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c not found: ID does not exist" containerID="747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c" Jul 14 20:19:16.002230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.002184 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c"} err="failed to get container status \"747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c\": rpc error: code = NotFound desc = could not find container \"747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c\": container with ID starting with 747fd88c707b7874712e3c18db2dec2a3b76cb2eee700685cc5651c400c4118c not found: ID does not exist" Jul 14 20:19:16.011663 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.011642 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l"] Jul 14 20:19:16.015956 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.015935 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ad8efdd2-openshift-default-785fdf9f47-zg78l"] Jul 14 20:19:16.830588 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.830557 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:16.894673 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.894644 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:16.984895 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:16.984844 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:17.036321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:17.036290 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" path="/var/lib/kubelet/pods/c76db8bb-2de9-4279-b327-46c4122a4e74/volumes" Jul 14 20:19:17.049311 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:17.049281 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:17.174438 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:17.174409 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:23.402555 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:23.402518 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:23.472198 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:23.472153 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:30.342486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.342393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:30.627927 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.627892 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq"] Jul 14 20:19:30.628260 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628247 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" Jul 14 20:19:30.628309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628262 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" Jul 14 20:19:30.628309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628273 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" Jul 14 20:19:30.628309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628278 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" Jul 14 20:19:30.628309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628296 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08948fc9-44c6-4260-8b25-179dc20568e5" containerName="maas-api" Jul 14 20:19:30.628309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628301 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="08948fc9-44c6-4260-8b25-179dc20568e5" containerName="maas-api" Jul 14 20:19:30.628480 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628357 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" Jul 14 20:19:30.628480 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.628365 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="08948fc9-44c6-4260-8b25-179dc20568e5" containerName="maas-api" Jul 14 20:19:30.632889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.632873 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.635661 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.635639 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4d0ddec0-openshift-default-dockercfg-nzk4j\"" Jul 14 20:19:30.643368 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.643348 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq"] Jul 14 20:19:30.651377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.651352 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:30.712889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.712858 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-data\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713043 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.712897 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-credential-socket\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713043 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.712919 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-certs\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713043 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.713020 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-podinfo\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713182 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.713050 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-socket\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713182 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.713089 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istiod-ca-cert\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713182 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.713107 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7kz\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-kube-api-access-zv7kz\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713280 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.713173 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-token\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.713280 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.713218 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-envoy\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814114 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814025 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-data\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814303 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814138 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-credential-socket\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814303 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814169 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-certs\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814303 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814223 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-podinfo\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814303 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814249 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-socket\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814303 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814290 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istiod-ca-cert\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814523 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814319 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7kz\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-kube-api-access-zv7kz\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814523 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814385 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-token\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814523 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814500 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-data\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814602 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-envoy\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814696 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814605 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-credential-socket\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.814936 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814918 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-certs\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.815033 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.814953 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-socket\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.815033 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.815004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istiod-ca-cert\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.816670 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.816644 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-envoy\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.816769 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.816753 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-podinfo\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.823279 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.823257 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-token\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.823534 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.823513 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7kz\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-kube-api-access-zv7kz\") pod \"e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.944924 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.944830 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:30.969252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:30.969217 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:31.086542 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:31.086514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq"] Jul 14 20:19:31.088409 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:19:31.088381 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2743503_083b_4c2e_9a5c_83ea096e4ccb.slice/crio-7f488024bba9e1414504d5821a22512dddf295235a0b93cea78b159259aa4972 WatchSource:0}: Error finding container 7f488024bba9e1414504d5821a22512dddf295235a0b93cea78b159259aa4972: Status 404 returned error can't find the container with id 7f488024bba9e1414504d5821a22512dddf295235a0b93cea78b159259aa4972 Jul 14 20:19:31.090508 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:31.090478 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:19:31.090593 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:31.090546 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:19:31.090593 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:31.090576 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:19:32.066147 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:32.066103 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" event={"ID":"a2743503-083b-4c2e-9a5c-83ea096e4ccb","Type":"ContainerStarted","Data":"c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442"} Jul 14 20:19:32.066623 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:32.066165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" event={"ID":"a2743503-083b-4c2e-9a5c-83ea096e4ccb","Type":"ContainerStarted","Data":"7f488024bba9e1414504d5821a22512dddf295235a0b93cea78b159259aa4972"} Jul 14 20:19:32.090185 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:32.090134 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podStartSLOduration=2.090113777 podStartE2EDuration="2.090113777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:19:32.089258493 +0000 UTC m=+1977.537884942" watchObservedRunningTime="2026-07-14 20:19:32.090113777 +0000 UTC m=+1977.538740223" Jul 14 20:19:32.946156 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:32.946114 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:32.951303 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:32.951274 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:33.072598 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:33.072561 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:33.073582 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:33.073563 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:34.000675 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.000634 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:34.086397 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.086364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:34.092016 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.091988 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v"] Jul 14 20:19:34.092435 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.092421 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76db8bb-2de9-4279-b327-46c4122a4e74" containerName="istio-proxy" Jul 14 20:19:34.095518 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.095502 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.098287 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.098266 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4d0ddec0-serving-cert\"" Jul 14 20:19:34.106012 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.105983 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v"] Jul 14 20:19:34.141122 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.141091 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30564649-2e6d-4153-aed6-d386d9b29834-maas-api-tls\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.141314 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.141147 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30564649-2e6d-4153-aed6-d386d9b29834-openshift-service-ca\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.141314 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.141210 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpwk\" (UniqueName: \"kubernetes.io/projected/30564649-2e6d-4153-aed6-d386d9b29834-kube-api-access-smpwk\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.241812 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.241778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpwk\" (UniqueName: \"kubernetes.io/projected/30564649-2e6d-4153-aed6-d386d9b29834-kube-api-access-smpwk\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.241996 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.241879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30564649-2e6d-4153-aed6-d386d9b29834-maas-api-tls\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.241996 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.241923 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30564649-2e6d-4153-aed6-d386d9b29834-openshift-service-ca\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.242633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.242611 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30564649-2e6d-4153-aed6-d386d9b29834-openshift-service-ca\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.244287 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.244270 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30564649-2e6d-4153-aed6-d386d9b29834-maas-api-tls\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.264359 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.264273 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpwk\" (UniqueName: \"kubernetes.io/projected/30564649-2e6d-4153-aed6-d386d9b29834-kube-api-access-smpwk\") pod \"maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.346443 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.346409 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:34.407922 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.407876 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:34.535245 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.535169 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:34.557371 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.557342 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v"] Jul 14 20:19:34.558410 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:19:34.558381 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30564649_2e6d_4153_aed6_d386d9b29834.slice/crio-cdf9275ab2329b5114cd21938175f88a837cf3b4ebf1c0c808ec13b664402c96 WatchSource:0}: Error finding container cdf9275ab2329b5114cd21938175f88a837cf3b4ebf1c0c808ec13b664402c96: Status 404 returned error can't find the container with id cdf9275ab2329b5114cd21938175f88a837cf3b4ebf1c0c808ec13b664402c96 Jul 14 20:19:34.837197 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.837114 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:34.943292 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:34.943262 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:35.084884 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:35.084851 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" event={"ID":"30564649-2e6d-4153-aed6-d386d9b29834","Type":"ContainerStarted","Data":"31c6bd1d4039906a03cc3a4e6209e5bc629129e8726255b0d488e32e7b76a9e7"} Jul 14 20:19:35.085055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:35.084913 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" event={"ID":"30564649-2e6d-4153-aed6-d386d9b29834","Type":"ContainerStarted","Data":"cdf9275ab2329b5114cd21938175f88a837cf3b4ebf1c0c808ec13b664402c96"} Jul 14 20:19:35.085207 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:35.085181 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:35.113326 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:35.113212 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" podStartSLOduration=0.707202965 podStartE2EDuration="1.113189196s" totalImagesPullingTime="405.986231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:19:35.113127863 +0000 UTC m=+1980.561754312" watchObservedRunningTime="2026-07-14 20:19:35.113189196 +0000 UTC m=+1980.561815646" Jul 14 20:19:35.177903 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:35.177867 2553 scope.go:117] "RemoveContainer" containerID="dd5ee63067acecb08bb559feca9d5d4d15884c9c31dba9a462ac0ae5b1082ff6" Jul 14 20:19:38.492314 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:38.492282 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:41.097476 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:41.097443 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:19:48.310351 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:48.310318 2553 patch_prober.go:28] interesting pod/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:57586->10.134.0.54:15021: read: connection reset by peer" start-of-body= Jul 14 20:19:48.310775 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:48.310387 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:57586->10.134.0.54:15021: read: connection reset by peer" Jul 14 20:19:49.154548 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:49.154517 2553 generic.go:356] "Generic (PLEG): container finished" podID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerID="c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442" exitCode=0 Jul 14 20:19:49.154721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:49.154591 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" event={"ID":"a2743503-083b-4c2e-9a5c-83ea096e4ccb","Type":"ContainerDied","Data":"c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442"} Jul 14 20:19:49.155019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:49.155004 2553 scope.go:117] "RemoveContainer" containerID="c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442" Jul 14 20:19:49.156006 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:49.155982 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:19:49.156082 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:49.156047 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:19:49.156124 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:49.156104 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:19:50.161818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:50.161784 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" event={"ID":"a2743503-083b-4c2e-9a5c-83ea096e4ccb","Type":"ContainerStarted","Data":"1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10"} Jul 14 20:19:50.945789 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:50.945737 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:50.946008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:50.945979 2553 patch_prober.go:28] interesting pod/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:19:50.946093 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:50.946039 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:19:51.945414 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:51.945376 2553 patch_prober.go:28] interesting pod/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:19:51.945809 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:51.945439 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:19:52.420355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:52.420325 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:19:52.946225 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:52.946188 2553 patch_prober.go:28] interesting pod/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:19:52.946615 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:52.946245 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:19:53.945468 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:53.945424 2553 patch_prober.go:28] interesting pod/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:19:53.945685 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:53.945485 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:19:54.948802 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:54.948765 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:54.949304 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:54.949167 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:19:54.949781 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:19:54.949761 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:20:03.238163 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:03.238131 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:03.338767 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:03.338733 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:03.469114 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:03.469054 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:03.570138 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:03.570034 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:17.514513 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:17.514469 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:17.617198 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:17.617158 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:27.236753 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.236719 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v"] Jul 14 20:20:27.237219 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.236987 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" podUID="30564649-2e6d-4153-aed6-d386d9b29834" containerName="maas-api" containerID="cri-o://31c6bd1d4039906a03cc3a4e6209e5bc629129e8726255b0d488e32e7b76a9e7" gracePeriod=30 Jul 14 20:20:27.260785 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.260758 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:27.364743 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.364695 2553 generic.go:356] "Generic (PLEG): container finished" podID="30564649-2e6d-4153-aed6-d386d9b29834" containerID="31c6bd1d4039906a03cc3a4e6209e5bc629129e8726255b0d488e32e7b76a9e7" exitCode=0 Jul 14 20:20:27.364903 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.364771 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" event={"ID":"30564649-2e6d-4153-aed6-d386d9b29834","Type":"ContainerDied","Data":"31c6bd1d4039906a03cc3a4e6209e5bc629129e8726255b0d488e32e7b76a9e7"} Jul 14 20:20:27.491319 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.491250 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:20:27.632508 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.632475 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smpwk\" (UniqueName: \"kubernetes.io/projected/30564649-2e6d-4153-aed6-d386d9b29834-kube-api-access-smpwk\") pod \"30564649-2e6d-4153-aed6-d386d9b29834\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " Jul 14 20:20:27.632686 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.632529 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30564649-2e6d-4153-aed6-d386d9b29834-openshift-service-ca\") pod \"30564649-2e6d-4153-aed6-d386d9b29834\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " Jul 14 20:20:27.632686 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.632575 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30564649-2e6d-4153-aed6-d386d9b29834-maas-api-tls\") pod \"30564649-2e6d-4153-aed6-d386d9b29834\" (UID: \"30564649-2e6d-4153-aed6-d386d9b29834\") " Jul 14 20:20:27.632979 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.632950 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30564649-2e6d-4153-aed6-d386d9b29834-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30564649-2e6d-4153-aed6-d386d9b29834" (UID: "30564649-2e6d-4153-aed6-d386d9b29834"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:20:27.634645 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.634618 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30564649-2e6d-4153-aed6-d386d9b29834-kube-api-access-smpwk" (OuterVolumeSpecName: "kube-api-access-smpwk") pod "30564649-2e6d-4153-aed6-d386d9b29834" (UID: "30564649-2e6d-4153-aed6-d386d9b29834"). InnerVolumeSpecName "kube-api-access-smpwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:20:27.634806 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.634785 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30564649-2e6d-4153-aed6-d386d9b29834-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30564649-2e6d-4153-aed6-d386d9b29834" (UID: "30564649-2e6d-4153-aed6-d386d9b29834"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:20:27.733859 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.733823 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smpwk\" (UniqueName: \"kubernetes.io/projected/30564649-2e6d-4153-aed6-d386d9b29834-kube-api-access-smpwk\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:20:27.733859 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.733851 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30564649-2e6d-4153-aed6-d386d9b29834-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:20:27.733859 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:27.733863 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30564649-2e6d-4153-aed6-d386d9b29834-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:20:28.371311 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:28.371287 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" Jul 14 20:20:28.371715 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:28.371322 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v" event={"ID":"30564649-2e6d-4153-aed6-d386d9b29834","Type":"ContainerDied","Data":"cdf9275ab2329b5114cd21938175f88a837cf3b4ebf1c0c808ec13b664402c96"} Jul 14 20:20:28.371715 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:28.371381 2553 scope.go:117] "RemoveContainer" containerID="31c6bd1d4039906a03cc3a4e6209e5bc629129e8726255b0d488e32e7b76a9e7" Jul 14 20:20:28.393979 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:28.393949 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v"] Jul 14 20:20:28.401226 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:28.401202 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d0ddec0-7dc486d57c-tjf5v"] Jul 14 20:20:29.036597 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:29.036565 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30564649-2e6d-4153-aed6-d386d9b29834" path="/var/lib/kubelet/pods/30564649-2e6d-4153-aed6-d386d9b29834/volumes" Jul 14 20:20:31.854200 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:31.854163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:36.929474 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:36.929439 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:50.487744 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:50.487707 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:20:50.543503 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:20:50.543471 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:14.269716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:14.269641 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:14.538093 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:14.537972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:20.572197 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:20.572160 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:35.106424 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:35.106385 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:21:35.108379 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:35.108347 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:35.112594 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:35.112569 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:21:35.205109 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:35.205056 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:44.157848 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:44.157810 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq"] Jul 14 20:21:44.158386 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:44.158088 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" containerID="cri-o://1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10" gracePeriod=30 Jul 14 20:21:44.182736 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:44.182701 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:44.378011 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:44.377978 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:45.412965 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:45.412926 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:45.512240 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:45.512181 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:45.612696 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:45.612660 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:46.272783 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:46.272750 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:46.371621 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:46.371576 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:46.471681 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:46.471649 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:47.816199 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:47.816160 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:49.412973 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.412951 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:21:49.553134 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553025 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-podinfo\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553134 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553093 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-token\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553134 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553112 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-envoy\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553158 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-data\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553188 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-socket\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553261 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istiod-ca-cert\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553294 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-certs\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553348 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-credential-socket\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553450 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv7kz\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-kube-api-access-zv7kz\") pod \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\" (UID: \"a2743503-083b-4c2e-9a5c-83ea096e4ccb\") " Jul 14 20:21:49.553827 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553545 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:21:49.553827 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553683 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:21:49.553827 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553809 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.553827 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553827 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.554042 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553917 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:21:49.554042 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.553974 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:21:49.554190 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.554097 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:21:49.555327 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.555304 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:21:49.555459 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.555395 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:21:49.555727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.555703 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:21:49.555821 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.555736 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-kube-api-access-zv7kz" (OuterVolumeSpecName: "kube-api-access-zv7kz") pod "a2743503-083b-4c2e-9a5c-83ea096e4ccb" (UID: "a2743503-083b-4c2e-9a5c-83ea096e4ccb"). InnerVolumeSpecName "kube-api-access-zv7kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:21:49.655052 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655011 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.655052 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655046 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv7kz\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-kube-api-access-zv7kz\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.655052 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655056 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.655307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655091 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.655307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655103 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.655307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655116 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.655307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.655127 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2743503-083b-4c2e-9a5c-83ea096e4ccb-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:21:49.808475 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.808390 2553 generic.go:356] "Generic (PLEG): container finished" podID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerID="1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10" exitCode=0 Jul 14 20:21:49.808475 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.808446 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" Jul 14 20:21:49.808674 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.808470 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" event={"ID":"a2743503-083b-4c2e-9a5c-83ea096e4ccb","Type":"ContainerDied","Data":"1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10"} Jul 14 20:21:49.808674 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.808521 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq" event={"ID":"a2743503-083b-4c2e-9a5c-83ea096e4ccb","Type":"ContainerDied","Data":"7f488024bba9e1414504d5821a22512dddf295235a0b93cea78b159259aa4972"} Jul 14 20:21:49.808674 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.808545 2553 scope.go:117] "RemoveContainer" containerID="1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10" Jul 14 20:21:49.817964 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.817944 2553 scope.go:117] "RemoveContainer" containerID="c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442" Jul 14 20:21:49.825889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.825869 2553 scope.go:117] "RemoveContainer" containerID="1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10" Jul 14 20:21:49.826130 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:21:49.826113 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10\": container with ID starting with 1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10 not found: ID does not exist" containerID="1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10" Jul 14 20:21:49.826174 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.826142 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10"} err="failed to get container status \"1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10\": rpc error: code = NotFound desc = could not find container \"1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10\": container with ID starting with 1343df08367ca232b5b232fa64bfb3c37f9e7f603dc9e107652059a5fc898e10 not found: ID does not exist" Jul 14 20:21:49.826174 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.826165 2553 scope.go:117] "RemoveContainer" containerID="c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442" Jul 14 20:21:49.826369 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:21:49.826354 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442\": container with ID starting with c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442 not found: ID does not exist" containerID="c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442" Jul 14 20:21:49.826418 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.826372 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442"} err="failed to get container status \"c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442\": rpc error: code = NotFound desc = could not find container \"c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442\": container with ID starting with c3257962d68763adfa9d1ec39160bd931c212e481df7dade591b2d87c4ac6442 not found: ID does not exist" Jul 14 20:21:49.833459 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.833439 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq"] Jul 14 20:21:49.837720 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:49.837701 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4d0ddec0-openshift-default-78fdbdcdc9-zmwhq"] Jul 14 20:21:50.458391 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:50.458355 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:50.651200 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:50.651165 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:51.036895 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:51.036866 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" path="/var/lib/kubelet/pods/a2743503-083b-4c2e-9a5c-83ea096e4ccb/volumes" Jul 14 20:21:53.772212 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:53.772174 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:53.834763 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:53.834729 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:21:54.429678 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:21:54.429644 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:00.711599 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:00.711554 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:00.823477 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:00.823360 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:00.912013 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:00.911960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:11.407625 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:11.407568 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:11.762502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:11.762421 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:32.807830 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:32.807736 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:38.799226 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:38.799190 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:38.865450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:38.865418 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:39.214673 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.214627 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:39.488088 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.487993 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g"] Jul 14 20:22:39.488378 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488365 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" Jul 14 20:22:39.488432 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488379 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" Jul 14 20:22:39.488432 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488391 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30564649-2e6d-4153-aed6-d386d9b29834" containerName="maas-api" Jul 14 20:22:39.488432 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488398 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="30564649-2e6d-4153-aed6-d386d9b29834" containerName="maas-api" Jul 14 20:22:39.488527 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488461 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" Jul 14 20:22:39.488527 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488469 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" Jul 14 20:22:39.488527 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.488476 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="30564649-2e6d-4153-aed6-d386d9b29834" containerName="maas-api" Jul 14 20:22:39.491514 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.491498 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.493969 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.493953 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1c67fa-openshift-default-dockercfg-6s9t7\"" Jul 14 20:22:39.503992 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.503968 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g"] Jul 14 20:22:39.509043 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.509020 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:39.593385 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593348 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-token\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593385 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593381 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istiod-ca-cert\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593408 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-data\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593448 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-credential-socket\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593480 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-socket\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593499 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-certs\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593593 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-podinfo\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593633 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593631 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4vp\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-kube-api-access-zl4vp\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.593887 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.593676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-envoy\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695035 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.694996 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-envoy\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695126 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-token\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695156 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istiod-ca-cert\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695194 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-data\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695233 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-credential-socket\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695267 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-socket\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695576 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695484 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-certs\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695576 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695572 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-podinfo\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695576 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695598 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4vp\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-kube-api-access-zl4vp\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695835 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695636 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-data\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695835 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695702 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-credential-socket\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695835 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695783 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-socket\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.695835 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.695802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-certs\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.696139 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.696117 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istiod-ca-cert\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.697425 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.697404 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-envoy\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.698151 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.698134 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-podinfo\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.707198 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.707177 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-token\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.707915 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.707893 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4vp\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-kube-api-access-zl4vp\") pod \"e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.794920 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.794834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:39.802904 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.802884 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:39.937550 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.937518 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g"] Jul 14 20:22:39.939035 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:22:39.939013 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc798f6e5_e750_440a_a70a_4c3ffd4b543e.slice/crio-b7549b62e1a8b8121793e37d949fec611fd124e34903587d1014ca1f923faa4d WatchSource:0}: Error finding container b7549b62e1a8b8121793e37d949fec611fd124e34903587d1014ca1f923faa4d: Status 404 returned error can't find the container with id b7549b62e1a8b8121793e37d949fec611fd124e34903587d1014ca1f923faa4d Jul 14 20:22:39.941144 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.941114 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:39.941232 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.941171 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:39.941232 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:39.941202 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:40.066904 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:40.066802 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" event={"ID":"c798f6e5-e750-440a-a70a-4c3ffd4b543e","Type":"ContainerStarted","Data":"99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4"} Jul 14 20:22:40.066904 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:40.066864 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" event={"ID":"c798f6e5-e750-440a-a70a-4c3ffd4b543e","Type":"ContainerStarted","Data":"b7549b62e1a8b8121793e37d949fec611fd124e34903587d1014ca1f923faa4d"} Jul 14 20:22:40.087320 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:40.087265 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podStartSLOduration=1.087245745 podStartE2EDuration="1.087245745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:22:40.086210802 +0000 UTC m=+2165.534837261" watchObservedRunningTime="2026-07-14 20:22:40.087245745 +0000 UTC m=+2165.535872191" Jul 14 20:22:40.803859 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:40.803816 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:40.809155 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:40.809126 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:41.072497 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:41.072417 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:41.073612 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:41.073590 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:42.052648 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.052611 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:42.149170 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.149134 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:42.246546 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.246513 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj"] Jul 14 20:22:42.246880 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.246868 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" Jul 14 20:22:42.246928 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.246881 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2743503-083b-4c2e-9a5c-83ea096e4ccb" containerName="istio-proxy" Jul 14 20:22:42.250021 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.250001 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.252370 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.252350 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1c67fa-serving-cert\"" Jul 14 20:22:42.264463 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.264436 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj"] Jul 14 20:22:42.286402 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.286364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:42.420224 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.420184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6577d211-4758-4bb9-8e53-06e18db5d7c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.420413 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.420244 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577d211-4758-4bb9-8e53-06e18db5d7c3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.420413 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.420266 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59gw\" (UniqueName: \"kubernetes.io/projected/6577d211-4758-4bb9-8e53-06e18db5d7c3-kube-api-access-z59gw\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.521557 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.521517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6577d211-4758-4bb9-8e53-06e18db5d7c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.521731 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.521584 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577d211-4758-4bb9-8e53-06e18db5d7c3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.521731 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.521610 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59gw\" (UniqueName: \"kubernetes.io/projected/6577d211-4758-4bb9-8e53-06e18db5d7c3-kube-api-access-z59gw\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.522334 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.522309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577d211-4758-4bb9-8e53-06e18db5d7c3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.523888 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.523866 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6577d211-4758-4bb9-8e53-06e18db5d7c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.530327 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.530308 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59gw\" (UniqueName: \"kubernetes.io/projected/6577d211-4758-4bb9-8e53-06e18db5d7c3-kube-api-access-z59gw\") pod \"maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.537721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.537691 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:42.560734 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.560705 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:42.696742 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.696716 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj"] Jul 14 20:22:42.698405 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:22:42.698381 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6577d211_4758_4bb9_8e53_06e18db5d7c3.slice/crio-5395f8e64a15cc6c7eb9dbd69234b810f1094de788a90783acfbe98b53b29eb0 WatchSource:0}: Error finding container 5395f8e64a15cc6c7eb9dbd69234b810f1094de788a90783acfbe98b53b29eb0: Status 404 returned error can't find the container with id 5395f8e64a15cc6c7eb9dbd69234b810f1094de788a90783acfbe98b53b29eb0 Jul 14 20:22:42.958092 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:42.957977 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:43.057617 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:43.057589 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:43.085892 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:43.085854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" event={"ID":"6577d211-4758-4bb9-8e53-06e18db5d7c3","Type":"ContainerStarted","Data":"5395f8e64a15cc6c7eb9dbd69234b810f1094de788a90783acfbe98b53b29eb0"} Jul 14 20:22:44.094200 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:44.094166 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" event={"ID":"6577d211-4758-4bb9-8e53-06e18db5d7c3","Type":"ContainerStarted","Data":"7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a"} Jul 14 20:22:44.094612 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:44.094235 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:44.112212 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:44.112163 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" podStartSLOduration=1.76196361 podStartE2EDuration="2.112145936s" totalImagesPullingTime="350.182326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:22:44.111100249 +0000 UTC m=+2169.559726696" watchObservedRunningTime="2026-07-14 20:22:44.112145936 +0000 UTC m=+2169.560772412" Jul 14 20:22:46.580715 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:46.580679 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:47.017187 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.017152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:47.236474 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.236441 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp"] Jul 14 20:22:47.240703 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.240685 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.243730 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.243709 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2e7e45-openshift-default-dockercfg-9n7z8\"" Jul 14 20:22:47.252168 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.252143 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp"] Jul 14 20:22:47.265849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.265826 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:47.365908 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.365823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-data\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.365908 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.365861 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-podinfo\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366123 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.365957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbmq\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-kube-api-access-xsbmq\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366123 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.366024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-certs\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366123 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.366088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-token\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366239 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.366134 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istiod-ca-cert\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366239 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.366152 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-credential-socket\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366239 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.366191 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-socket\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.366239 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.366210 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-envoy\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbmq\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-kube-api-access-xsbmq\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467684 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-certs\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467684 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467536 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-token\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467684 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467572 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istiod-ca-cert\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467684 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467594 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-credential-socket\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467684 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467615 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-socket\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.467684 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-envoy\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468087 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-data\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468087 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467711 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-podinfo\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468087 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.467992 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-certs\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468087 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.468053 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-credential-socket\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468288 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.468132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-socket\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468288 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.468206 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-data\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.468447 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.468425 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istiod-ca-cert\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.470035 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.470014 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-envoy\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.470321 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.470301 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-podinfo\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.476096 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.476050 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-token\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.476185 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.476165 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbmq\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-kube-api-access-xsbmq\") pod \"e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.552969 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.552933 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:47.648963 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.648921 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:47.699415 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.699383 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp"] Jul 14 20:22:47.701294 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:22:47.701265 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9096e4c7_f76c_4e15_8c82_372aedc2e09d.slice/crio-8526a03b80ceacca6f5dc0c77eb11c552608b338b72240c2d07b528ba5933cee WatchSource:0}: Error finding container 8526a03b80ceacca6f5dc0c77eb11c552608b338b72240c2d07b528ba5933cee: Status 404 returned error can't find the container with id 8526a03b80ceacca6f5dc0c77eb11c552608b338b72240c2d07b528ba5933cee Jul 14 20:22:47.703378 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.703341 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:47.703470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.703413 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:47.703470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:47.703448 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:48.117589 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:48.117498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" event={"ID":"9096e4c7-f76c-4e15-8c82-372aedc2e09d","Type":"ContainerStarted","Data":"6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374"} Jul 14 20:22:48.117589 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:48.117554 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" event={"ID":"9096e4c7-f76c-4e15-8c82-372aedc2e09d","Type":"ContainerStarted","Data":"8526a03b80ceacca6f5dc0c77eb11c552608b338b72240c2d07b528ba5933cee"} Jul 14 20:22:48.138402 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:48.138353 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podStartSLOduration=1.138337073 podStartE2EDuration="1.138337073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:22:48.13741371 +0000 UTC m=+2173.586040156" watchObservedRunningTime="2026-07-14 20:22:48.138337073 +0000 UTC m=+2173.586963518" Jul 14 20:22:48.553415 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:48.553376 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:48.558355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:48.558330 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:49.122840 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:49.122805 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:49.123777 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:49.123759 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:22:50.083777 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.083734 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:50.105013 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.104984 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:22:50.169146 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.169113 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq"] Jul 14 20:22:50.170494 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.170461 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:50.173018 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.172995 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.175333 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.175311 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2e7e45-serving-cert\"" Jul 14 20:22:50.191368 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.191321 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq"] Jul 14 20:22:50.240038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.240002 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:50.297751 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.297714 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb2e083-e189-4c5a-86b1-effbd73500d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.297751 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.297752 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggs6z\" (UniqueName: \"kubernetes.io/projected/4fb2e083-e189-4c5a-86b1-effbd73500d4-kube-api-access-ggs6z\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.297983 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.297925 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb2e083-e189-4c5a-86b1-effbd73500d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.338727 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.338622 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:50.398752 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.398695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb2e083-e189-4c5a-86b1-effbd73500d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.398924 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.398857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb2e083-e189-4c5a-86b1-effbd73500d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.398924 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.398888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggs6z\" (UniqueName: \"kubernetes.io/projected/4fb2e083-e189-4c5a-86b1-effbd73500d4-kube-api-access-ggs6z\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.399967 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.399920 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb2e083-e189-4c5a-86b1-effbd73500d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.402450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.402413 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb2e083-e189-4c5a-86b1-effbd73500d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.409199 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.409178 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggs6z\" (UniqueName: \"kubernetes.io/projected/4fb2e083-e189-4c5a-86b1-effbd73500d4-kube-api-access-ggs6z\") pod \"maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.490540 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.490497 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:50.628479 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.628445 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq"] Jul 14 20:22:50.630303 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:22:50.630273 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb2e083_e189_4c5a_86b1_effbd73500d4.slice/crio-c96decf27c3e822a867d01c02aebaac216f7293bd5eb772377feb8c804ebbf33 WatchSource:0}: Error finding container c96decf27c3e822a867d01c02aebaac216f7293bd5eb772377feb8c804ebbf33: Status 404 returned error can't find the container with id c96decf27c3e822a867d01c02aebaac216f7293bd5eb772377feb8c804ebbf33 Jul 14 20:22:50.774157 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.774124 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:50.871483 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:50.871407 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:51.134869 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:51.134837 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" event={"ID":"4fb2e083-e189-4c5a-86b1-effbd73500d4","Type":"ContainerStarted","Data":"c96decf27c3e822a867d01c02aebaac216f7293bd5eb772377feb8c804ebbf33"} Jul 14 20:22:52.142193 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:52.142158 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" event={"ID":"4fb2e083-e189-4c5a-86b1-effbd73500d4","Type":"ContainerStarted","Data":"2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792"} Jul 14 20:22:52.142589 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:52.142296 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:52.160973 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:52.160908 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" podStartSLOduration=1.6551130729999999 podStartE2EDuration="2.160892334s" totalImagesPullingTime="505.779261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:22:52.158669866 +0000 UTC m=+2177.607296315" watchObservedRunningTime="2026-07-14 20:22:52.160892334 +0000 UTC m=+2177.609518781" Jul 14 20:22:54.634898 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:54.634869 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:22:57.113246 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:57.113206 2553 patch_prober.go:28] interesting pod/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:22:57.113616 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:57.113288 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:22:58.153193 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.153164 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:22:58.185230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.185132 2553 generic.go:356] "Generic (PLEG): container finished" podID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerID="99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4" exitCode=0 Jul 14 20:22:58.185994 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.185960 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" event={"ID":"c798f6e5-e750-440a-a70a-4c3ffd4b543e","Type":"ContainerDied","Data":"99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4"} Jul 14 20:22:58.186489 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.186462 2553 scope.go:117] "RemoveContainer" containerID="99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4" Jul 14 20:22:58.187980 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.187940 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:58.188094 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.188035 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:58.188170 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:58.188101 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:22:59.197408 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:59.197368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" event={"ID":"c798f6e5-e750-440a-a70a-4c3ffd4b543e","Type":"ContainerStarted","Data":"d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6"} Jul 14 20:22:59.803777 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:59.803731 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:22:59.804103 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:59.803846 2553 patch_prober.go:28] interesting pod/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 14 20:22:59.804298 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:22:59.804264 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 14 20:23:00.804134 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:00.804094 2553 patch_prober.go:28] interesting pod/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 14 20:23:00.805178 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:00.804720 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 14 20:23:01.804423 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:01.804330 2553 patch_prober.go:28] interesting pod/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 14 20:23:01.804423 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:01.804388 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 14 20:23:02.804253 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:02.804206 2553 patch_prober.go:28] interesting pod/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 14 20:23:02.804433 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:02.804276 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 14 20:23:03.807010 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:03.806972 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:23:03.807542 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:03.807244 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:23:03.807953 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:03.807929 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:23:04.619779 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:04.619739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:04.649667 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:04.649616 2553 patch_prober.go:28] interesting pod/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:55858->10.134.0.58:15021: read: connection reset by peer" start-of-body= Jul 14 20:23:04.649814 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:04.649710 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:55858->10.134.0.58:15021: read: connection reset by peer" Jul 14 20:23:04.719910 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:04.719882 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:04.820832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:04.820758 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:05.241121 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:05.241085 2553 generic.go:356] "Generic (PLEG): container finished" podID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerID="6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374" exitCode=0 Jul 14 20:23:05.241317 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:05.241158 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" event={"ID":"9096e4c7-f76c-4e15-8c82-372aedc2e09d","Type":"ContainerDied","Data":"6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374"} Jul 14 20:23:05.241663 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:05.241647 2553 scope.go:117] "RemoveContainer" containerID="6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374" Jul 14 20:23:05.242650 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:05.242619 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:23:05.242725 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:05.242692 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:23:05.242765 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:05.242725 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:23:06.248222 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:06.248185 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" event={"ID":"9096e4c7-f76c-4e15-8c82-372aedc2e09d","Type":"ContainerStarted","Data":"5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d"} Jul 14 20:23:06.553577 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:06.553496 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:23:06.553730 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:06.553688 2553 patch_prober.go:28] interesting pod/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 14 20:23:06.553774 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:06.553742 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 14 20:23:07.554049 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:07.554013 2553 patch_prober.go:28] interesting pod/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 14 20:23:07.554504 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:07.554090 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 14 20:23:08.554340 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:08.554295 2553 patch_prober.go:28] interesting pod/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 14 20:23:08.554722 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:08.554389 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 14 20:23:09.553964 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:09.553929 2553 patch_prober.go:28] interesting pod/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 14 20:23:09.554167 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:09.553985 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 14 20:23:10.556971 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:10.556941 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:23:10.557355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:10.557282 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:23:10.557975 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:10.557957 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:23:12.825849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:12.825815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:12.926169 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:12.926122 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:25.617758 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:25.617717 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:48.232752 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:48.232694 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:49.130983 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:49.130950 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:49.774816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:49.774767 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:49.875606 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:49.875551 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:55.005572 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:55.005528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:23:59.771452 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:59.771369 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj"] Jul 14 20:23:59.771895 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:59.771632 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" podUID="6577d211-4758-4bb9-8e53-06e18db5d7c3" containerName="maas-api" containerID="cri-o://7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a" gracePeriod=30 Jul 14 20:23:59.797597 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:23:59.797571 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:00.017462 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.017437 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:24:00.168850 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.168815 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577d211-4758-4bb9-8e53-06e18db5d7c3-openshift-service-ca\") pod \"6577d211-4758-4bb9-8e53-06e18db5d7c3\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " Jul 14 20:24:00.168850 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.168851 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z59gw\" (UniqueName: \"kubernetes.io/projected/6577d211-4758-4bb9-8e53-06e18db5d7c3-kube-api-access-z59gw\") pod \"6577d211-4758-4bb9-8e53-06e18db5d7c3\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " Jul 14 20:24:00.169139 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.168875 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6577d211-4758-4bb9-8e53-06e18db5d7c3-maas-api-tls\") pod \"6577d211-4758-4bb9-8e53-06e18db5d7c3\" (UID: \"6577d211-4758-4bb9-8e53-06e18db5d7c3\") " Jul 14 20:24:00.169281 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.169252 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6577d211-4758-4bb9-8e53-06e18db5d7c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6577d211-4758-4bb9-8e53-06e18db5d7c3" (UID: "6577d211-4758-4bb9-8e53-06e18db5d7c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:24:00.169433 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.169400 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577d211-4758-4bb9-8e53-06e18db5d7c3-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:00.171048 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.171017 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6577d211-4758-4bb9-8e53-06e18db5d7c3-kube-api-access-z59gw" (OuterVolumeSpecName: "kube-api-access-z59gw") pod "6577d211-4758-4bb9-8e53-06e18db5d7c3" (UID: "6577d211-4758-4bb9-8e53-06e18db5d7c3"). InnerVolumeSpecName "kube-api-access-z59gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:00.171167 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.171039 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6577d211-4758-4bb9-8e53-06e18db5d7c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6577d211-4758-4bb9-8e53-06e18db5d7c3" (UID: "6577d211-4758-4bb9-8e53-06e18db5d7c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:24:00.270717 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.270680 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59gw\" (UniqueName: \"kubernetes.io/projected/6577d211-4758-4bb9-8e53-06e18db5d7c3-kube-api-access-z59gw\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:00.270717 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.270710 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6577d211-4758-4bb9-8e53-06e18db5d7c3-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:00.541309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.541224 2553 generic.go:356] "Generic (PLEG): container finished" podID="6577d211-4758-4bb9-8e53-06e18db5d7c3" containerID="7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a" exitCode=0 Jul 14 20:24:00.541309 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.541273 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" Jul 14 20:24:00.541487 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.541314 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" event={"ID":"6577d211-4758-4bb9-8e53-06e18db5d7c3","Type":"ContainerDied","Data":"7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a"} Jul 14 20:24:00.541487 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.541368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj" event={"ID":"6577d211-4758-4bb9-8e53-06e18db5d7c3","Type":"ContainerDied","Data":"5395f8e64a15cc6c7eb9dbd69234b810f1094de788a90783acfbe98b53b29eb0"} Jul 14 20:24:00.541487 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.541396 2553 scope.go:117] "RemoveContainer" containerID="7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a" Jul 14 20:24:00.550522 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.550505 2553 scope.go:117] "RemoveContainer" containerID="7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a" Jul 14 20:24:00.550776 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:00.550756 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a\": container with ID starting with 7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a not found: ID does not exist" containerID="7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a" Jul 14 20:24:00.550838 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.550789 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a"} err="failed to get container status \"7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a\": rpc error: code = NotFound desc = could not find container \"7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a\": container with ID starting with 7f7ec537cadee25910c7eb87bf06eb63640cf48ff07c1ee0325eb0a328aeab5a not found: ID does not exist" Jul 14 20:24:00.564167 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.564139 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj"] Jul 14 20:24:00.570516 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:00.570480 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c67fa-75f9b7f64f-cnsfj"] Jul 14 20:24:01.036037 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:01.036004 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6577d211-4758-4bb9-8e53-06e18db5d7c3" path="/var/lib/kubelet/pods/6577d211-4758-4bb9-8e53-06e18db5d7c3/volumes" Jul 14 20:24:04.342025 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:04.341991 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:10.458776 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:10.458742 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:10.559723 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:10.559662 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:15.041885 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:15.041854 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g"] Jul 14 20:24:15.042360 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:15.042114 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" containerID="cri-o://d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6" gracePeriod=30 Jul 14 20:24:15.064427 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:15.064381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:15.295975 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:15.295862 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:15.933250 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:15.933215 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:16.033421 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:16.033390 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:18.809456 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:18.809414 2553 patch_prober.go:28] interesting pod/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:24:18.809839 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:18.809485 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:24:20.289677 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.289653 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:24:20.361421 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361335 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istiod-ca-cert\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361421 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361368 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-socket\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361421 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361413 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-data\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361436 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl4vp\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-kube-api-access-zl4vp\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361451 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-envoy\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361472 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-certs\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361490 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-credential-socket\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361535 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-token\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361558 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-podinfo\") pod \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\" (UID: \"c798f6e5-e750-440a-a70a-4c3ffd4b543e\") " Jul 14 20:24:20.361985 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361872 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:24:20.361985 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361873 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:20.362115 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.361991 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:20.362115 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.362056 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:20.362304 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.362280 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:20.363933 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.363903 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:20.364088 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.363977 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:20.364228 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.364204 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-kube-api-access-zl4vp" (OuterVolumeSpecName: "kube-api-access-zl4vp") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "kube-api-access-zl4vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:20.364272 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.364233 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c798f6e5-e750-440a-a70a-4c3ffd4b543e" (UID: "c798f6e5-e750-440a-a70a-4c3ffd4b543e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:24:20.462661 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462633 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462661 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462658 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462661 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462666 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462912 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462674 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462912 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462684 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462912 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462692 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4vp\" (UniqueName: \"kubernetes.io/projected/c798f6e5-e750-440a-a70a-4c3ffd4b543e-kube-api-access-zl4vp\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462912 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462700 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462912 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462709 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.462912 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.462720 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c798f6e5-e750-440a-a70a-4c3ffd4b543e-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:20.651136 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.651103 2553 generic.go:356] "Generic (PLEG): container finished" podID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerID="d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6" exitCode=0 Jul 14 20:24:20.651305 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.651146 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" Jul 14 20:24:20.651305 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.651179 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" event={"ID":"c798f6e5-e750-440a-a70a-4c3ffd4b543e","Type":"ContainerDied","Data":"d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6"} Jul 14 20:24:20.651305 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.651234 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g" event={"ID":"c798f6e5-e750-440a-a70a-4c3ffd4b543e","Type":"ContainerDied","Data":"b7549b62e1a8b8121793e37d949fec611fd124e34903587d1014ca1f923faa4d"} Jul 14 20:24:20.651305 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.651261 2553 scope.go:117] "RemoveContainer" containerID="d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6" Jul 14 20:24:20.660531 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.660512 2553 scope.go:117] "RemoveContainer" containerID="99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4" Jul 14 20:24:20.668906 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.668885 2553 scope.go:117] "RemoveContainer" containerID="d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6" Jul 14 20:24:20.669170 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:20.669149 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6\": container with ID starting with d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6 not found: ID does not exist" containerID="d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6" Jul 14 20:24:20.669214 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.669180 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6"} err="failed to get container status \"d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6\": rpc error: code = NotFound desc = could not find container \"d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6\": container with ID starting with d0b19a1365d363b3c01278f44da0826594a238810aaf6466b9c34d6852290cd6 not found: ID does not exist" Jul 14 20:24:20.669214 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.669199 2553 scope.go:117] "RemoveContainer" containerID="99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4" Jul 14 20:24:20.669428 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:20.669413 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4\": container with ID starting with 99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4 not found: ID does not exist" containerID="99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4" Jul 14 20:24:20.669470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.669431 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4"} err="failed to get container status \"99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4\": rpc error: code = NotFound desc = could not find container \"99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4\": container with ID starting with 99a782a2a70cfaf462056d4bd87d814b077214f44169b18b5a91f88578d477b4 not found: ID does not exist" Jul 14 20:24:20.673867 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.673845 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g"] Jul 14 20:24:20.678830 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:20.678809 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1c67fa-openshift-default-668b8c66d9-ft74g"] Jul 14 20:24:21.037339 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:21.037266 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" path="/var/lib/kubelet/pods/c798f6e5-e750-440a-a70a-4c3ffd4b543e/volumes" Jul 14 20:24:25.866663 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:25.866626 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq"] Jul 14 20:24:25.867170 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:25.866894 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" podUID="4fb2e083-e189-4c5a-86b1-effbd73500d4" containerName="maas-api" containerID="cri-o://2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792" gracePeriod=30 Jul 14 20:24:25.895634 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:25.895606 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:26.117966 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.117908 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:24:26.214644 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.214609 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb2e083-e189-4c5a-86b1-effbd73500d4-openshift-service-ca\") pod \"4fb2e083-e189-4c5a-86b1-effbd73500d4\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " Jul 14 20:24:26.214814 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.214672 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb2e083-e189-4c5a-86b1-effbd73500d4-maas-api-tls\") pod \"4fb2e083-e189-4c5a-86b1-effbd73500d4\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " Jul 14 20:24:26.214814 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.214710 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggs6z\" (UniqueName: \"kubernetes.io/projected/4fb2e083-e189-4c5a-86b1-effbd73500d4-kube-api-access-ggs6z\") pod \"4fb2e083-e189-4c5a-86b1-effbd73500d4\" (UID: \"4fb2e083-e189-4c5a-86b1-effbd73500d4\") " Jul 14 20:24:26.215076 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.215033 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb2e083-e189-4c5a-86b1-effbd73500d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4fb2e083-e189-4c5a-86b1-effbd73500d4" (UID: "4fb2e083-e189-4c5a-86b1-effbd73500d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:24:26.216832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.216808 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb2e083-e189-4c5a-86b1-effbd73500d4-kube-api-access-ggs6z" (OuterVolumeSpecName: "kube-api-access-ggs6z") pod "4fb2e083-e189-4c5a-86b1-effbd73500d4" (UID: "4fb2e083-e189-4c5a-86b1-effbd73500d4"). InnerVolumeSpecName "kube-api-access-ggs6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:26.216926 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.216902 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb2e083-e189-4c5a-86b1-effbd73500d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4fb2e083-e189-4c5a-86b1-effbd73500d4" (UID: "4fb2e083-e189-4c5a-86b1-effbd73500d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:24:26.316270 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.316234 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggs6z\" (UniqueName: \"kubernetes.io/projected/4fb2e083-e189-4c5a-86b1-effbd73500d4-kube-api-access-ggs6z\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:26.316270 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.316267 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb2e083-e189-4c5a-86b1-effbd73500d4-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:26.316270 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.316280 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb2e083-e189-4c5a-86b1-effbd73500d4-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:26.589916 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.589868 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:26.684551 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.684515 2553 generic.go:356] "Generic (PLEG): container finished" podID="4fb2e083-e189-4c5a-86b1-effbd73500d4" containerID="2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792" exitCode=0 Jul 14 20:24:26.684740 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.684566 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" Jul 14 20:24:26.684740 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.684597 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" event={"ID":"4fb2e083-e189-4c5a-86b1-effbd73500d4","Type":"ContainerDied","Data":"2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792"} Jul 14 20:24:26.684740 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.684650 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq" event={"ID":"4fb2e083-e189-4c5a-86b1-effbd73500d4","Type":"ContainerDied","Data":"c96decf27c3e822a867d01c02aebaac216f7293bd5eb772377feb8c804ebbf33"} Jul 14 20:24:26.684740 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.684673 2553 scope.go:117] "RemoveContainer" containerID="2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792" Jul 14 20:24:26.694329 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.694315 2553 scope.go:117] "RemoveContainer" containerID="2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792" Jul 14 20:24:26.694577 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:26.694559 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792\": container with ID starting with 2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792 not found: ID does not exist" containerID="2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792" Jul 14 20:24:26.694615 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.694586 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792"} err="failed to get container status \"2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792\": rpc error: code = NotFound desc = could not find container \"2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792\": container with ID starting with 2ab1c4e4bc3c66d377cc4be1b968e15f47c697c9ba68af0a4cead627ae564792 not found: ID does not exist" Jul 14 20:24:26.708113 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.708089 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq"] Jul 14 20:24:26.711052 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:26.711030 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e7e45-7c6cfdcf78-nnmkq"] Jul 14 20:24:27.037384 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:27.037352 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb2e083-e189-4c5a-86b1-effbd73500d4" path="/var/lib/kubelet/pods/4fb2e083-e189-4c5a-86b1-effbd73500d4/volumes" Jul 14 20:24:30.498664 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:30.498628 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:36.617475 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:36.617439 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:36.717556 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:36.717524 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:41.213249 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:41.213216 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp"] Jul 14 20:24:41.213672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:41.213419 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" containerID="cri-o://5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d" gracePeriod=30 Jul 14 20:24:41.230345 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:41.230317 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:41.465180 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:41.465079 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:41.842937 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:41.842827 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:42.129989 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.129953 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd"] Jul 14 20:24:42.130411 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130393 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6577d211-4758-4bb9-8e53-06e18db5d7c3" containerName="maas-api" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130412 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6577d211-4758-4bb9-8e53-06e18db5d7c3" containerName="maas-api" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130432 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130440 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130460 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130469 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130478 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb2e083-e189-4c5a-86b1-effbd73500d4" containerName="maas-api" Jul 14 20:24:42.130505 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130486 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb2e083-e189-4c5a-86b1-effbd73500d4" containerName="maas-api" Jul 14 20:24:42.130849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130568 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" Jul 14 20:24:42.130849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130580 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6577d211-4758-4bb9-8e53-06e18db5d7c3" containerName="maas-api" Jul 14 20:24:42.130849 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.130592 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb2e083-e189-4c5a-86b1-effbd73500d4" containerName="maas-api" Jul 14 20:24:42.133796 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.133775 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.136451 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.136427 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-285387-openshift-default-dockercfg-zj6qb\"" Jul 14 20:24:42.144720 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.144701 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd"] Jul 14 20:24:42.159103 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.159076 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:42.257923 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.257887 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-socket\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.257923 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.257922 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-credential-socket\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.257946 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-token\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.257991 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-podinfo\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.258032 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-certs\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.258071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsszg\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-kube-api-access-gsszg\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.258098 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istiod-ca-cert\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.258174 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-data\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.258406 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.258225 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-envoy\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.352737 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.352707 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:42.359210 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359185 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-envoy\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359279 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-socket\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359307 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-credential-socket\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359338 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-token\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359370 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-podinfo\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359601 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359418 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-certs\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359601 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359454 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsszg\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-kube-api-access-gsszg\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359601 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istiod-ca-cert\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359601 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359576 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-data\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359825 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359685 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-socket\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.359874 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359857 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-certs\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.360002 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.359982 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-data\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.360110 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.360074 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-credential-socket\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.360518 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.360495 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istiod-ca-cert\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.361472 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.361451 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-envoy\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.361722 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.361705 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-podinfo\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.368225 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.368183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-token\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.368962 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.368939 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsszg\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-kube-api-access-gsszg\") pod \"e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.444808 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.444714 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:42.582441 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.582410 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd"] Jul 14 20:24:42.584407 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:24:42.584381 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680a878b_7ddc_44ed_b3f0_e7c4cd1f1638.slice/crio-8abdbc1f02fe5404357e0f052fad6e9419ff2d1d398ecf7d56a38f8c8b6e4df7 WatchSource:0}: Error finding container 8abdbc1f02fe5404357e0f052fad6e9419ff2d1d398ecf7d56a38f8c8b6e4df7: Status 404 returned error can't find the container with id 8abdbc1f02fe5404357e0f052fad6e9419ff2d1d398ecf7d56a38f8c8b6e4df7 Jul 14 20:24:42.586617 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.586583 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:42.586721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.586653 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:42.586721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.586690 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:42.768976 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.768877 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" event={"ID":"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638","Type":"ContainerStarted","Data":"848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de"} Jul 14 20:24:42.768976 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.768971 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" event={"ID":"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638","Type":"ContainerStarted","Data":"8abdbc1f02fe5404357e0f052fad6e9419ff2d1d398ecf7d56a38f8c8b6e4df7"} Jul 14 20:24:42.791597 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:42.791545 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podStartSLOduration=0.79152925 podStartE2EDuration="791.52925ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:24:42.788505374 +0000 UTC m=+2288.237131821" watchObservedRunningTime="2026-07-14 20:24:42.79152925 +0000 UTC m=+2288.240155696" Jul 14 20:24:43.445234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:43.445199 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:43.450207 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:43.450179 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:43.774402 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:43.774305 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:43.775345 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:43.775329 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:44.718609 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.718549 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:44.793577 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.793542 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc"] Jul 14 20:24:44.794206 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.794181 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c798f6e5-e750-440a-a70a-4c3ffd4b543e" containerName="istio-proxy" Jul 14 20:24:44.794796 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.794774 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:44.797711 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.797691 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.800214 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.800187 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-285387-serving-cert\"" Jul 14 20:24:44.808719 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.808696 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc"] Jul 14 20:24:44.859987 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.859953 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:44.884312 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.884271 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0087eca5-289a-496f-ad8d-2e921f688ae9-openshift-service-ca\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.884729 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.884695 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.884841 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.884796 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb6q\" (UniqueName: \"kubernetes.io/projected/0087eca5-289a-496f-ad8d-2e921f688ae9-kube-api-access-cfb6q\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.986460 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.986348 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.986460 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.986422 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb6q\" (UniqueName: \"kubernetes.io/projected/0087eca5-289a-496f-ad8d-2e921f688ae9-kube-api-access-cfb6q\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.986712 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:44.986525 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-285387-serving-cert: secret "maas-api-e2e-api-a-285387-serving-cert" not found Jul 14 20:24:44.986712 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.986545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0087eca5-289a-496f-ad8d-2e921f688ae9-openshift-service-ca\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:44.986712 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:44.986612 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls podName:0087eca5-289a-496f-ad8d-2e921f688ae9 nodeName:}" failed. No retries permitted until 2026-07-14 20:24:45.486594309 +0000 UTC m=+2290.935220733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls") pod "maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" (UID: "0087eca5-289a-496f-ad8d-2e921f688ae9") : secret "maas-api-e2e-api-a-285387-serving-cert" not found Jul 14 20:24:44.987153 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:44.987135 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0087eca5-289a-496f-ad8d-2e921f688ae9-openshift-service-ca\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:45.003340 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.003296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb6q\" (UniqueName: \"kubernetes.io/projected/0087eca5-289a-496f-ad8d-2e921f688ae9-kube-api-access-cfb6q\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:45.193853 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.193818 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:45.446923 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.446886 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:45.491195 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.491160 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:45.493644 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.493615 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls\") pod \"maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:45.542579 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.542539 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:45.709758 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.709669 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:45.850911 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.850884 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc"] Jul 14 20:24:45.852752 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:24:45.852727 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0087eca5_289a_496f_ad8d_2e921f688ae9.slice/crio-cadb53ad5295f2d86d1c656e53ceb6b95d33d259d7efa772d8f8bfc72fc3a16c WatchSource:0}: Error finding container cadb53ad5295f2d86d1c656e53ceb6b95d33d259d7efa772d8f8bfc72fc3a16c: Status 404 returned error can't find the container with id cadb53ad5295f2d86d1c656e53ceb6b95d33d259d7efa772d8f8bfc72fc3a16c Jul 14 20:24:45.854690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:45.854672 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:24:46.466116 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.466089 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:24:46.600983 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.600892 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-token\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.600983 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.600928 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-podinfo\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.600983 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.600970 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-credential-socket\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.600990 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsbmq\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-kube-api-access-xsbmq\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601016 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-socket\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601043 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istiod-ca-cert\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601082 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-certs\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601126 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-envoy\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601300 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601159 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-data\") pod \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\" (UID: \"9096e4c7-f76c-4e15-8c82-372aedc2e09d\") " Jul 14 20:24:46.601611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601374 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:46.601611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601449 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:46.601611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601509 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.601611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601523 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:24:46.601807 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.601678 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:46.602294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.602267 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:46.603396 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.603365 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:24:46.603538 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.603517 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-kube-api-access-xsbmq" (OuterVolumeSpecName: "kube-api-access-xsbmq") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "kube-api-access-xsbmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:46.603609 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.603550 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:46.603661 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.603612 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9096e4c7-f76c-4e15-8c82-372aedc2e09d" (UID: "9096e4c7-f76c-4e15-8c82-372aedc2e09d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:46.702236 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702200 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702236 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702228 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702236 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702237 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702236 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702244 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702254 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsbmq\" (UniqueName: \"kubernetes.io/projected/9096e4c7-f76c-4e15-8c82-372aedc2e09d-kube-api-access-xsbmq\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702263 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702272 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9096e4c7-f76c-4e15-8c82-372aedc2e09d-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.702506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.702280 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9096e4c7-f76c-4e15-8c82-372aedc2e09d-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:24:46.800818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.800779 2553 generic.go:356] "Generic (PLEG): container finished" podID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerID="5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d" exitCode=0 Jul 14 20:24:46.801007 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.800835 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" Jul 14 20:24:46.801007 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.800858 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" event={"ID":"9096e4c7-f76c-4e15-8c82-372aedc2e09d","Type":"ContainerDied","Data":"5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d"} Jul 14 20:24:46.801007 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.800936 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp" event={"ID":"9096e4c7-f76c-4e15-8c82-372aedc2e09d","Type":"ContainerDied","Data":"8526a03b80ceacca6f5dc0c77eb11c552608b338b72240c2d07b528ba5933cee"} Jul 14 20:24:46.801007 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.800971 2553 scope.go:117] "RemoveContainer" containerID="5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d" Jul 14 20:24:46.802610 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.802582 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" event={"ID":"0087eca5-289a-496f-ad8d-2e921f688ae9","Type":"ContainerStarted","Data":"bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61"} Jul 14 20:24:46.802731 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.802643 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:46.802731 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.802668 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" event={"ID":"0087eca5-289a-496f-ad8d-2e921f688ae9","Type":"ContainerStarted","Data":"cadb53ad5295f2d86d1c656e53ceb6b95d33d259d7efa772d8f8bfc72fc3a16c"} Jul 14 20:24:46.810741 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.810716 2553 scope.go:117] "RemoveContainer" containerID="6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374" Jul 14 20:24:46.818784 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.818765 2553 scope.go:117] "RemoveContainer" containerID="5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d" Jul 14 20:24:46.819037 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:46.819019 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d\": container with ID starting with 5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d not found: ID does not exist" containerID="5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d" Jul 14 20:24:46.819090 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.819046 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d"} err="failed to get container status \"5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d\": rpc error: code = NotFound desc = could not find container \"5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d\": container with ID starting with 5e1050674db3bd7b120becbbd59bf9e9bae2726f040a82214ba281e876ad8a1d not found: ID does not exist" Jul 14 20:24:46.819090 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.819075 2553 scope.go:117] "RemoveContainer" containerID="6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374" Jul 14 20:24:46.819286 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:46.819270 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374\": container with ID starting with 6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374 not found: ID does not exist" containerID="6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374" Jul 14 20:24:46.819323 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.819290 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374"} err="failed to get container status \"6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374\": rpc error: code = NotFound desc = could not find container \"6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374\": container with ID starting with 6b6f5355b13d7f63dc108ed9a17446e6d22c67db546cddf3f0458fa133c2f374 not found: ID does not exist" Jul 14 20:24:46.823048 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.823007 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" podStartSLOduration=2.422326148 podStartE2EDuration="2.822994348s" totalImagesPullingTime="400.6682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:24:46.820934195 +0000 UTC m=+2292.269560665" watchObservedRunningTime="2026-07-14 20:24:46.822994348 +0000 UTC m=+2292.271620793" Jul 14 20:24:46.837345 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.837322 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp"] Jul 14 20:24:46.841507 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:46.841484 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2e7e45-openshift-default-7d9687d5ff-v7krp"] Jul 14 20:24:47.036597 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:47.036564 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" path="/var/lib/kubelet/pods/9096e4c7-f76c-4e15-8c82-372aedc2e09d/volumes" Jul 14 20:24:49.310580 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.310547 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:49.695151 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.695117 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:49.974032 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.973942 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl"] Jul 14 20:24:49.974407 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.974390 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" Jul 14 20:24:49.974471 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.974408 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" Jul 14 20:24:49.974471 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.974427 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" Jul 14 20:24:49.974471 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.974433 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" Jul 14 20:24:49.974637 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.974489 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" Jul 14 20:24:49.979324 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.979303 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:49.981657 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.981636 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d3ee05-openshift-default-dockercfg-r869x\"" Jul 14 20:24:49.992860 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:49.992838 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl"] Jul 14 20:24:50.003257 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.003229 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:50.133495 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133457 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-data\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133495 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133500 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-certs\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133522 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-podinfo\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133539 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-credential-socket\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133556 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-token\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133576 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-socket\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s829l\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-kube-api-access-s829l\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133647 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635c4981-2e76-44c6-a805-1a1649d9dbf5-istiod-ca-cert\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.133898 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.133714 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-envoy\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234602 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-envoy\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234602 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-data\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234786 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234605 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-certs\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234786 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234623 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-podinfo\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234786 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234641 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-credential-socket\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234786 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234656 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-token\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234991 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234799 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-socket\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234991 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s829l\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-kube-api-access-s829l\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.234991 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.234897 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635c4981-2e76-44c6-a805-1a1649d9dbf5-istiod-ca-cert\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.235189 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.235097 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-data\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.235189 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.235104 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-credential-socket\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.235295 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.235255 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-socket\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.235295 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.235262 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-certs\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.235832 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.235809 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635c4981-2e76-44c6-a805-1a1649d9dbf5-istiod-ca-cert\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.236901 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.236883 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-envoy\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.237194 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.237178 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-podinfo\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.243468 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.243451 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s829l\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-kube-api-access-s829l\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.243560 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.243539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-token\") pod \"e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.290293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.290240 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:50.434156 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.434132 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl"] Jul 14 20:24:50.435944 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:24:50.435916 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635c4981_2e76_44c6_a805_1a1649d9dbf5.slice/crio-f4ac85e2ddf404f08a97aa4c7aa7c9ee09581d2c0bb5b628e594efb5c42458d7 WatchSource:0}: Error finding container f4ac85e2ddf404f08a97aa4c7aa7c9ee09581d2c0bb5b628e594efb5c42458d7: Status 404 returned error can't find the container with id f4ac85e2ddf404f08a97aa4c7aa7c9ee09581d2c0bb5b628e594efb5c42458d7 Jul 14 20:24:50.438302 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.438271 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:50.438358 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.438339 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:50.438399 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.438369 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:50.721938 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.721876 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:50.827741 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.827656 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" event={"ID":"635c4981-2e76-44c6-a805-1a1649d9dbf5","Type":"ContainerStarted","Data":"f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff"} Jul 14 20:24:50.827741 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.827704 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" event={"ID":"635c4981-2e76-44c6-a805-1a1649d9dbf5","Type":"ContainerStarted","Data":"f4ac85e2ddf404f08a97aa4c7aa7c9ee09581d2c0bb5b628e594efb5c42458d7"} Jul 14 20:24:50.848888 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:50.848839 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podStartSLOduration=1.848821671 podStartE2EDuration="1.848821671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:24:50.846929061 +0000 UTC m=+2296.295555545" watchObservedRunningTime="2026-07-14 20:24:50.848821671 +0000 UTC m=+2296.297448117" Jul 14 20:24:51.291456 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:51.291410 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:51.296621 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:51.296594 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:51.833437 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:51.833406 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:51.834371 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:51.834351 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:24:52.815508 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:52.815471 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:24:52.894087 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:52.890501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:52.986088 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:52.982284 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn"] Jul 14 20:24:52.986088 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:52.982935 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9096e4c7-f76c-4e15-8c82-372aedc2e09d" containerName="istio-proxy" Jul 14 20:24:52.998157 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:52.998126 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.001873 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.001636 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d3ee05-serving-cert\"" Jul 14 20:24:53.004397 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.004151 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn"] Jul 14 20:24:53.039670 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.039630 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:53.063620 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.063586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.063778 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.063626 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.063778 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.063670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklr6\" (UniqueName: \"kubernetes.io/projected/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-kube-api-access-sklr6\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.164393 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.164355 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.164608 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.164401 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.164608 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.164429 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklr6\" (UniqueName: \"kubernetes.io/projected/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-kube-api-access-sklr6\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.164608 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:53.164500 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-serving-cert: secret "maas-api-e2e-api-b-d3ee05-serving-cert" not found Jul 14 20:24:53.164608 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:24:53.164573 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls podName:fb55b964-6c63-4b6c-90a3-43dca6f3aa0e nodeName:}" failed. No retries permitted until 2026-07-14 20:24:53.664552647 +0000 UTC m=+2299.113179071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls") pod "maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" (UID: "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e") : secret "maas-api-e2e-api-b-d3ee05-serving-cert" not found Jul 14 20:24:53.165316 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.165294 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.184035 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.184002 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklr6\" (UniqueName: \"kubernetes.io/projected/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-kube-api-access-sklr6\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.221306 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.221276 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:53.484262 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.484157 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:53.590236 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.590201 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:53.670571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.670525 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.673942 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.673907 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:53.690692 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.690638 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:53.913276 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:53.913241 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:54.092712 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:54.092672 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn"] Jul 14 20:24:54.098091 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:24:54.098043 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb55b964_6c63_4b6c_90a3_43dca6f3aa0e.slice/crio-40f93cc8717d707b5df4b049c7c308f5de9c20b486796c2e9fbdd6bbf270f32d WatchSource:0}: Error finding container 40f93cc8717d707b5df4b049c7c308f5de9c20b486796c2e9fbdd6bbf270f32d: Status 404 returned error can't find the container with id 40f93cc8717d707b5df4b049c7c308f5de9c20b486796c2e9fbdd6bbf270f32d Jul 14 20:24:54.857915 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:54.857870 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" event={"ID":"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e","Type":"ContainerStarted","Data":"332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146"} Jul 14 20:24:54.858104 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:54.857963 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:24:54.858104 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:54.857981 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" event={"ID":"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e","Type":"ContainerStarted","Data":"40f93cc8717d707b5df4b049c7c308f5de9c20b486796c2e9fbdd6bbf270f32d"} Jul 14 20:24:54.875458 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:54.875388 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" podStartSLOduration=2.2356954350000002 podStartE2EDuration="2.875365513s" totalImagesPullingTime="639.670078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:24:54.874335784 +0000 UTC m=+2300.322962235" watchObservedRunningTime="2026-07-14 20:24:54.875365513 +0000 UTC m=+2300.323991959" Jul 14 20:24:57.295029 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.294990 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:24:57.876770 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.876738 2553 generic.go:356] "Generic (PLEG): container finished" podID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerID="848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de" exitCode=0 Jul 14 20:24:57.876989 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.876813 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" event={"ID":"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638","Type":"ContainerDied","Data":"848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de"} Jul 14 20:24:57.877252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.877238 2553 scope.go:117] "RemoveContainer" containerID="848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de" Jul 14 20:24:57.878212 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.878184 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:57.878268 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.878255 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:57.878304 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:57.878286 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:24:58.775054 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:58.775023 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:58.884316 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:58.884283 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" event={"ID":"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638","Type":"ContainerStarted","Data":"f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc"} Jul 14 20:24:59.445582 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:59.445548 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:24:59.446641 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:59.446614 2553 patch_prober.go:28] interesting pod/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 14 20:24:59.446767 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:24:59.446665 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 14 20:25:00.445829 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:00.445789 2553 patch_prober.go:28] interesting pod/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 14 20:25:00.446240 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:00.445849 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 14 20:25:00.880224 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:00.880189 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:25:01.445613 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:01.445571 2553 patch_prober.go:28] interesting pod/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 14 20:25:01.445798 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:01.445636 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 14 20:25:02.445288 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:02.445249 2553 patch_prober.go:28] interesting pod/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 14 20:25:02.445772 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:02.445316 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 14 20:25:03.449591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:03.449560 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:25:03.449993 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:03.449814 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:25:03.450487 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:03.450467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:25:04.275218 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:04.275179 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:04.371638 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:04.371604 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:06.090766 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:06.090714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:06.207404 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:06.207366 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:07.874261 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:07.874222 2553 patch_prober.go:28] interesting pod/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:25:07.874677 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:07.874283 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:25:08.952103 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:08.952047 2553 generic.go:356] "Generic (PLEG): container finished" podID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerID="f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff" exitCode=0 Jul 14 20:25:08.952551 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:08.952119 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" event={"ID":"635c4981-2e76-44c6-a805-1a1649d9dbf5","Type":"ContainerDied","Data":"f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff"} Jul 14 20:25:08.952551 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:08.952533 2553 scope.go:117] "RemoveContainer" containerID="f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff" Jul 14 20:25:08.953590 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:08.953559 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:25:08.953660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:08.953647 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:25:08.953696 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:08.953687 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:25:09.959631 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:09.959594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" event={"ID":"635c4981-2e76-44c6-a805-1a1649d9dbf5","Type":"ContainerStarted","Data":"a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f"} Jul 14 20:25:10.291319 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:10.291210 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:25:10.291539 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:10.291519 2553 patch_prober.go:28] interesting pod/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 20:25:10.291613 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:10.291578 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 20:25:11.290772 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:11.290721 2553 patch_prober.go:28] interesting pod/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 20:25:11.291313 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:11.290804 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 20:25:12.291607 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:12.291568 2553 patch_prober.go:28] interesting pod/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 20:25:12.292020 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:12.291626 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 20:25:13.291234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:13.291194 2553 patch_prober.go:28] interesting pod/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 14 20:25:13.291412 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:13.291261 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 14 20:25:14.293965 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:14.293935 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:25:14.294480 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:14.294279 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:25:14.294818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:14.294799 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:25:15.480646 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:15.480610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:15.580846 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:15.580808 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:15.736413 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:15.736320 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc"] Jul 14 20:25:15.736609 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:15.736564 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" podUID="0087eca5-289a-496f-ad8d-2e921f688ae9" containerName="maas-api" containerID="cri-o://bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61" gracePeriod=30 Jul 14 20:25:15.769116 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:15.769057 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:16.005344 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.005312 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:25:16.014549 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.014521 2553 generic.go:356] "Generic (PLEG): container finished" podID="0087eca5-289a-496f-ad8d-2e921f688ae9" containerID="bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61" exitCode=0 Jul 14 20:25:16.014686 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.014555 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" Jul 14 20:25:16.014686 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.014609 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" event={"ID":"0087eca5-289a-496f-ad8d-2e921f688ae9","Type":"ContainerDied","Data":"bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61"} Jul 14 20:25:16.014686 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.014662 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc" event={"ID":"0087eca5-289a-496f-ad8d-2e921f688ae9","Type":"ContainerDied","Data":"cadb53ad5295f2d86d1c656e53ceb6b95d33d259d7efa772d8f8bfc72fc3a16c"} Jul 14 20:25:16.014860 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.014693 2553 scope.go:117] "RemoveContainer" containerID="bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61" Jul 14 20:25:16.024804 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.024476 2553 scope.go:117] "RemoveContainer" containerID="bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61" Jul 14 20:25:16.024804 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:25:16.024788 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61\": container with ID starting with bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61 not found: ID does not exist" containerID="bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61" Jul 14 20:25:16.024986 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.024823 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61"} err="failed to get container status \"bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61\": rpc error: code = NotFound desc = could not find container \"bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61\": container with ID starting with bb3ec64bf84b85529c314fe90a636798f955697aac30bcf78bda7f1441f49b61 not found: ID does not exist" Jul 14 20:25:16.091397 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.091362 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls\") pod \"0087eca5-289a-496f-ad8d-2e921f688ae9\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " Jul 14 20:25:16.091584 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.091459 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0087eca5-289a-496f-ad8d-2e921f688ae9-openshift-service-ca\") pod \"0087eca5-289a-496f-ad8d-2e921f688ae9\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " Jul 14 20:25:16.091584 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.091509 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfb6q\" (UniqueName: \"kubernetes.io/projected/0087eca5-289a-496f-ad8d-2e921f688ae9-kube-api-access-cfb6q\") pod \"0087eca5-289a-496f-ad8d-2e921f688ae9\" (UID: \"0087eca5-289a-496f-ad8d-2e921f688ae9\") " Jul 14 20:25:16.091988 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.091956 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0087eca5-289a-496f-ad8d-2e921f688ae9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0087eca5-289a-496f-ad8d-2e921f688ae9" (UID: "0087eca5-289a-496f-ad8d-2e921f688ae9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:25:16.093557 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.093535 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0087eca5-289a-496f-ad8d-2e921f688ae9" (UID: "0087eca5-289a-496f-ad8d-2e921f688ae9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:25:16.093632 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.093600 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0087eca5-289a-496f-ad8d-2e921f688ae9-kube-api-access-cfb6q" (OuterVolumeSpecName: "kube-api-access-cfb6q") pod "0087eca5-289a-496f-ad8d-2e921f688ae9" (UID: "0087eca5-289a-496f-ad8d-2e921f688ae9"). InnerVolumeSpecName "kube-api-access-cfb6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:25:16.192959 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.192924 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0087eca5-289a-496f-ad8d-2e921f688ae9-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:16.192959 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.192953 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0087eca5-289a-496f-ad8d-2e921f688ae9-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:16.192959 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.192964 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfb6q\" (UniqueName: \"kubernetes.io/projected/0087eca5-289a-496f-ad8d-2e921f688ae9-kube-api-access-cfb6q\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:16.337272 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.337238 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc"] Jul 14 20:25:16.341541 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:16.341511 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-285387-5fdf7f746f-tzrwc"] Jul 14 20:25:17.035571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:17.035530 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0087eca5-289a-496f-ad8d-2e921f688ae9" path="/var/lib/kubelet/pods/0087eca5-289a-496f-ad8d-2e921f688ae9/volumes" Jul 14 20:25:20.549808 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:20.549773 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:26.618164 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:26.618051 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:26.718169 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:26.718134 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:31.208970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:31.208934 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd"] Jul 14 20:25:31.209422 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:31.209206 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" containerID="cri-o://f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc" gracePeriod=30 Jul 14 20:25:31.230606 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:31.230577 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:31.449401 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:31.449367 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:33.452480 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:33.452438 2553 patch_prober.go:28] interesting pod/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 14 20:25:33.452879 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:33.452516 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:25:36.472234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.472209 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:25:36.577162 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577041 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsszg\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-kube-api-access-gsszg\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577162 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577105 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istiod-ca-cert\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577162 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577164 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-token\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577194 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-socket\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577218 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-data\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577260 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-podinfo\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577339 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-credential-socket\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577388 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-certs\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577450 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577428 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-envoy\") pod \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\" (UID: \"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638\") " Jul 14 20:25:36.577743 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577512 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:25:36.577743 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577582 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:25:36.577743 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577667 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:25:36.577743 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577701 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:25:36.577941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577785 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.577941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577799 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.577941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577813 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.577941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.577827 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.578246 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.578222 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-data" (OuterVolumeSpecName: "istio-data") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:25:36.579581 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.579559 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:25:36.579581 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.579567 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-token" (OuterVolumeSpecName: "istio-token") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:25:36.579796 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.579773 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:25:36.579854 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.579799 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-kube-api-access-gsszg" (OuterVolumeSpecName: "kube-api-access-gsszg") pod "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" (UID: "680a878b-7ddc-44ed-b3f0-e7c4cd1f1638"). InnerVolumeSpecName "kube-api-access-gsszg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:25:36.678496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.678443 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.678496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.678490 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsszg\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-kube-api-access-gsszg\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.678496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.678501 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.678496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.678510 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:36.678496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:36.678518 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:37.133905 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.133868 2553 generic.go:356] "Generic (PLEG): container finished" podID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerID="f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc" exitCode=0 Jul 14 20:25:37.134107 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.133930 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" Jul 14 20:25:37.134107 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.133932 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" event={"ID":"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638","Type":"ContainerDied","Data":"f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc"} Jul 14 20:25:37.134107 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.134075 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd" event={"ID":"680a878b-7ddc-44ed-b3f0-e7c4cd1f1638","Type":"ContainerDied","Data":"8abdbc1f02fe5404357e0f052fad6e9419ff2d1d398ecf7d56a38f8c8b6e4df7"} Jul 14 20:25:37.134107 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.134107 2553 scope.go:117] "RemoveContainer" containerID="f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc" Jul 14 20:25:37.142691 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.142671 2553 scope.go:117] "RemoveContainer" containerID="848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de" Jul 14 20:25:37.151324 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.150919 2553 scope.go:117] "RemoveContainer" containerID="f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc" Jul 14 20:25:37.152104 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:25:37.151400 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc\": container with ID starting with f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc not found: ID does not exist" containerID="f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc" Jul 14 20:25:37.152104 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.151443 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc"} err="failed to get container status \"f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc\": rpc error: code = NotFound desc = could not find container \"f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc\": container with ID starting with f6fc799215e4bb56ca6a44dd33b7f700fdf7730307cd5f92d7319184604971fc not found: ID does not exist" Jul 14 20:25:37.152104 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.151472 2553 scope.go:117] "RemoveContainer" containerID="848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de" Jul 14 20:25:37.152104 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:25:37.151735 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de\": container with ID starting with 848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de not found: ID does not exist" containerID="848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de" Jul 14 20:25:37.152104 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.151764 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de"} err="failed to get container status \"848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de\": rpc error: code = NotFound desc = could not find container \"848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de\": container with ID starting with 848f33ff1ca736063c5479ba9f3c730139670ecf73d4297a28493bcf0cc2a8de not found: ID does not exist" Jul 14 20:25:37.154114 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.154093 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd"] Jul 14 20:25:37.157895 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:37.157875 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-285387-openshift-default-85cddcff6f-ffjwd"] Jul 14 20:25:39.036700 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:39.036668 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" path="/var/lib/kubelet/pods/680a878b-7ddc-44ed-b3f0-e7c4cd1f1638/volumes" Jul 14 20:25:42.174120 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.174080 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn"] Jul 14 20:25:42.174611 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.174362 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" podUID="fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" containerName="maas-api" containerID="cri-o://332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146" gracePeriod=30 Jul 14 20:25:42.206545 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.206513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:42.421355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.421330 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:25:42.535524 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.535421 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-openshift-service-ca\") pod \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " Jul 14 20:25:42.535706 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.535559 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls\") pod \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " Jul 14 20:25:42.535706 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.535602 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sklr6\" (UniqueName: \"kubernetes.io/projected/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-kube-api-access-sklr6\") pod \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\" (UID: \"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e\") " Jul 14 20:25:42.535829 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.535801 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" (UID: "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:25:42.535895 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.535878 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:42.537756 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.537734 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" (UID: "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:25:42.537812 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.537736 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-kube-api-access-sklr6" (OuterVolumeSpecName: "kube-api-access-sklr6") pod "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" (UID: "fb55b964-6c63-4b6c-90a3-43dca6f3aa0e"). InnerVolumeSpecName "kube-api-access-sklr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:25:42.637222 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.637183 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:42.637222 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:42.637214 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklr6\" (UniqueName: \"kubernetes.io/projected/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e-kube-api-access-sklr6\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:25:43.181634 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.181600 2553 generic.go:356] "Generic (PLEG): container finished" podID="fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" containerID="332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146" exitCode=0 Jul 14 20:25:43.182118 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.181659 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" Jul 14 20:25:43.182118 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.181681 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" event={"ID":"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e","Type":"ContainerDied","Data":"332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146"} Jul 14 20:25:43.182118 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.181736 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn" event={"ID":"fb55b964-6c63-4b6c-90a3-43dca6f3aa0e","Type":"ContainerDied","Data":"40f93cc8717d707b5df4b049c7c308f5de9c20b486796c2e9fbdd6bbf270f32d"} Jul 14 20:25:43.182118 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.181759 2553 scope.go:117] "RemoveContainer" containerID="332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146" Jul 14 20:25:43.193593 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.193574 2553 scope.go:117] "RemoveContainer" containerID="332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146" Jul 14 20:25:43.193909 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:25:43.193889 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146\": container with ID starting with 332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146 not found: ID does not exist" containerID="332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146" Jul 14 20:25:43.193965 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.193916 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146"} err="failed to get container status \"332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146\": rpc error: code = NotFound desc = could not find container \"332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146\": container with ID starting with 332e4e03a777e48f651ee174ccfa376359e199e50ad71d68a6758ab4f61a3146 not found: ID does not exist" Jul 14 20:25:43.200231 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.200200 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn"] Jul 14 20:25:43.207712 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:43.207686 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ee05-84d5d6fb9c-sw4xn"] Jul 14 20:25:45.037871 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:45.037831 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" path="/var/lib/kubelet/pods/fb55b964-6c63-4b6c-90a3-43dca6f3aa0e/volumes" Jul 14 20:25:46.662970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:46.662926 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:52.860343 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:52.860310 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:52.927220 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:52.927184 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:57.327603 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:57.327570 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl"] Jul 14 20:25:57.328078 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:57.327772 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" containerID="cri-o://a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f" gracePeriod=30 Jul 14 20:25:57.350843 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:57.350814 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:57.616464 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:57.616379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:57.952193 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:57.952154 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:58.260930 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.260840 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv"] Jul 14 20:25:58.261240 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261224 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" containerName="maas-api" Jul 14 20:25:58.261294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261241 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" containerName="maas-api" Jul 14 20:25:58.261294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261259 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" Jul 14 20:25:58.261294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261265 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" Jul 14 20:25:58.261294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261276 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" Jul 14 20:25:58.261294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261281 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" Jul 14 20:25:58.261294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261291 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0087eca5-289a-496f-ad8d-2e921f688ae9" containerName="maas-api" Jul 14 20:25:58.261525 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261298 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0087eca5-289a-496f-ad8d-2e921f688ae9" containerName="maas-api" Jul 14 20:25:58.261525 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261353 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" Jul 14 20:25:58.261525 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261361 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="680a878b-7ddc-44ed-b3f0-e7c4cd1f1638" containerName="istio-proxy" Jul 14 20:25:58.261525 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261369 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0087eca5-289a-496f-ad8d-2e921f688ae9" containerName="maas-api" Jul 14 20:25:58.261525 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.261377 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb55b964-6c63-4b6c-90a3-43dca6f3aa0e" containerName="maas-api" Jul 14 20:25:58.265838 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.265819 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.268499 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.268470 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fad346-openshift-default-dockercfg-m958g\"" Jul 14 20:25:58.277209 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.277185 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv"] Jul 14 20:25:58.375645 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375611 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-credential-socket\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.375645 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375641 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-data\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-podinfo\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375760 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgb2\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-kube-api-access-pfgb2\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375844 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istiod-ca-cert\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375872 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-token\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375933 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-socket\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.375970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-envoy\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.376108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.376016 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-certs\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.476785 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476748 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istiod-ca-cert\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.476785 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476793 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-token\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476815 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-socket\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-envoy\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476895 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-certs\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476923 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-credential-socket\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476939 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-data\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-podinfo\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477019 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.476993 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgb2\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-kube-api-access-pfgb2\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477412 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.477344 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-data\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477469 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.477408 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-credential-socket\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477469 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.477443 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-certs\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477538 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.477476 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-socket\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.477689 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.477671 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istiod-ca-cert\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.479268 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.479247 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-envoy\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.479521 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.479499 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-podinfo\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.486118 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.486057 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-token\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.486817 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.486796 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgb2\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-kube-api-access-pfgb2\") pod \"e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.499800 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.499772 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:58.577011 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.576918 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:58.712007 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.711964 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv"] Jul 14 20:25:58.714402 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:25:58.714358 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2265f6_3d86_4c3f_ae57_9c60d9ae368b.slice/crio-b68a18cd285848261b52bd3b07eacbdc348358a8a8d83c8855b28c362be9da18 WatchSource:0}: Error finding container b68a18cd285848261b52bd3b07eacbdc348358a8a8d83c8855b28c362be9da18: Status 404 returned error can't find the container with id b68a18cd285848261b52bd3b07eacbdc348358a8a8d83c8855b28c362be9da18 Jul 14 20:25:58.716432 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.716402 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:25:58.716503 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.716465 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:25:58.716503 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.716494 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:25:58.990713 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:58.990673 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:25:59.263082 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.262971 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" event={"ID":"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b","Type":"ContainerStarted","Data":"d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f"} Jul 14 20:25:59.263082 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.263030 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" event={"ID":"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b","Type":"ContainerStarted","Data":"b68a18cd285848261b52bd3b07eacbdc348358a8a8d83c8855b28c362be9da18"} Jul 14 20:25:59.285125 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.285041 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podStartSLOduration=1.285020487 podStartE2EDuration="1.285020487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:25:59.283008467 +0000 UTC m=+2364.731634937" watchObservedRunningTime="2026-07-14 20:25:59.285020487 +0000 UTC m=+2364.733646932" Jul 14 20:25:59.296808 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.296774 2553 patch_prober.go:28] interesting pod/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:25:59.296955 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.296828 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:25:59.578137 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.578018 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:25:59.583531 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:25:59.583503 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:00.269041 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:00.269012 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:00.270036 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:00.270017 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:01.246281 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.246245 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:01.325891 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.325860 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp"] Jul 14 20:26:01.329624 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.329603 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.332185 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.332164 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fad346-serving-cert\"" Jul 14 20:26:01.340424 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.340401 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp"] Jul 14 20:26:01.342195 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.342167 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:01.403393 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.403355 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/834f3bf9-b411-46a7-8fa2-e0adff2d883f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.403571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.403414 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhlf\" (UniqueName: \"kubernetes.io/projected/834f3bf9-b411-46a7-8fa2-e0adff2d883f-kube-api-access-2nhlf\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.403571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.403472 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.416123 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.416085 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:01.504578 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.504483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhlf\" (UniqueName: \"kubernetes.io/projected/834f3bf9-b411-46a7-8fa2-e0adff2d883f-kube-api-access-2nhlf\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.504578 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.504537 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.504781 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.504602 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/834f3bf9-b411-46a7-8fa2-e0adff2d883f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.504781 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:26:01.504692 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-serving-cert: secret "maas-api-e2e-shared-a-fad346-serving-cert" not found Jul 14 20:26:01.504781 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:26:01.504753 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls podName:834f3bf9-b411-46a7-8fa2-e0adff2d883f nodeName:}" failed. No retries permitted until 2026-07-14 20:26:02.004737064 +0000 UTC m=+2367.453363487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls") pod "maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" (UID: "834f3bf9-b411-46a7-8fa2-e0adff2d883f") : secret "maas-api-e2e-shared-a-fad346-serving-cert" not found Jul 14 20:26:01.505227 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.505210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/834f3bf9-b411-46a7-8fa2-e0adff2d883f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.515561 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.515540 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhlf\" (UniqueName: \"kubernetes.io/projected/834f3bf9-b411-46a7-8fa2-e0adff2d883f-kube-api-access-2nhlf\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:01.817056 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.816951 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:01.931110 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:01.931079 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:02.008831 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.008799 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:02.011294 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.011269 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:02.030041 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.030010 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:02.242164 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.242124 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:02.393463 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.393436 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp"] Jul 14 20:26:02.401028 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:26:02.400995 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834f3bf9_b411_46a7_8fa2_e0adff2d883f.slice/crio-e1a7570b728b867cf661979d5096ae92885e7536f55d67f894eecdc260a7e106 WatchSource:0}: Error finding container e1a7570b728b867cf661979d5096ae92885e7536f55d67f894eecdc260a7e106: Status 404 returned error can't find the container with id e1a7570b728b867cf661979d5096ae92885e7536f55d67f894eecdc260a7e106 Jul 14 20:26:02.578251 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.578222 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:26:02.613523 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613490 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-data\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613532 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-podinfo\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613615 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-socket\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613657 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-token\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613707 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613680 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635c4981-2e76-44c6-a805-1a1649d9dbf5-istiod-ca-cert\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613996 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613717 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-envoy\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613996 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613771 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-certs\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.613996 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.613949 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.614190 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614079 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635c4981-2e76-44c6-a805-1a1649d9dbf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:26:02.614253 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614229 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-credential-socket\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.614307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614258 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.614307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614277 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s829l\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-kube-api-access-s829l\") pod \"635c4981-2e76-44c6-a805-1a1649d9dbf5\" (UID: \"635c4981-2e76-44c6-a805-1a1649d9dbf5\") " Jul 14 20:26:02.614493 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614449 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.614568 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614527 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.614658 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614637 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635c4981-2e76-44c6-a805-1a1649d9dbf5-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.614720 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614663 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.614720 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614678 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.614720 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614694 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.614720 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.614707 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.616096 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.616038 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:26:02.616470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.616448 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:02.616592 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.616567 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.616592 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.616582 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-kube-api-access-s829l" (OuterVolumeSpecName: "kube-api-access-s829l") pod "635c4981-2e76-44c6-a805-1a1649d9dbf5" (UID: "635c4981-2e76-44c6-a805-1a1649d9dbf5"). InnerVolumeSpecName "kube-api-access-s829l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:02.715192 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.715155 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.715192 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.715186 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.715192 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.715195 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s829l\" (UniqueName: \"kubernetes.io/projected/635c4981-2e76-44c6-a805-1a1649d9dbf5-kube-api-access-s829l\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.715192 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:02.715203 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635c4981-2e76-44c6-a805-1a1649d9dbf5-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:26:03.286053 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.285944 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" event={"ID":"834f3bf9-b411-46a7-8fa2-e0adff2d883f","Type":"ContainerStarted","Data":"788ab0bd01074b9b1a50c0071ffc08c2b141d88a1af8247c85e1afaaebc8f2c6"} Jul 14 20:26:03.286053 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.286004 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" event={"ID":"834f3bf9-b411-46a7-8fa2-e0adff2d883f","Type":"ContainerStarted","Data":"e1a7570b728b867cf661979d5096ae92885e7536f55d67f894eecdc260a7e106"} Jul 14 20:26:03.286053 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.286051 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:03.287584 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.287562 2553 generic.go:356] "Generic (PLEG): container finished" podID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerID="a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f" exitCode=0 Jul 14 20:26:03.287716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.287604 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" Jul 14 20:26:03.287716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.287649 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" event={"ID":"635c4981-2e76-44c6-a805-1a1649d9dbf5","Type":"ContainerDied","Data":"a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f"} Jul 14 20:26:03.287716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.287708 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl" event={"ID":"635c4981-2e76-44c6-a805-1a1649d9dbf5","Type":"ContainerDied","Data":"f4ac85e2ddf404f08a97aa4c7aa7c9ee09581d2c0bb5b628e594efb5c42458d7"} Jul 14 20:26:03.287826 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.287738 2553 scope.go:117] "RemoveContainer" containerID="a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f" Jul 14 20:26:03.296630 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.296608 2553 scope.go:117] "RemoveContainer" containerID="f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff" Jul 14 20:26:03.304629 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.304613 2553 scope.go:117] "RemoveContainer" containerID="a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f" Jul 14 20:26:03.304899 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:26:03.304879 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f\": container with ID starting with a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f not found: ID does not exist" containerID="a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f" Jul 14 20:26:03.304953 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.304910 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f"} err="failed to get container status \"a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f\": rpc error: code = NotFound desc = could not find container \"a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f\": container with ID starting with a5004a4b6b56ae4dee11e698e94696de85747597a64416f907bd5cf63863713f not found: ID does not exist" Jul 14 20:26:03.304953 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.304929 2553 scope.go:117] "RemoveContainer" containerID="f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff" Jul 14 20:26:03.305215 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:26:03.305188 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff\": container with ID starting with f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff not found: ID does not exist" containerID="f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff" Jul 14 20:26:03.305260 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.305220 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff"} err="failed to get container status \"f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff\": rpc error: code = NotFound desc = could not find container \"f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff\": container with ID starting with f0eb72f1ab4ea3721b43d57bbbb1aa7a7ef0ca4967dcfa8c7f2deef40fc1b5ff not found: ID does not exist" Jul 14 20:26:03.311809 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.311766 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" podStartSLOduration=1.855233175 podStartE2EDuration="2.311751525s" totalImagesPullingTime="456.51835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:03.309783086 +0000 UTC m=+2368.758409569" watchObservedRunningTime="2026-07-14 20:26:03.311751525 +0000 UTC m=+2368.760377971" Jul 14 20:26:03.329264 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.329235 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl"] Jul 14 20:26:03.351874 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:03.351843 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d3ee05-openshift-default-7cc5b87f8f-d2gvl"] Jul 14 20:26:05.036408 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:05.036379 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" path="/var/lib/kubelet/pods/635c4981-2e76-44c6-a805-1a1649d9dbf5/volumes" Jul 14 20:26:05.915359 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:05.915318 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:06.105702 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.105667 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:06.367444 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.367338 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c"] Jul 14 20:26:06.368027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.368004 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" Jul 14 20:26:06.368027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.368026 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" Jul 14 20:26:06.368203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.368055 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" Jul 14 20:26:06.368203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.368079 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" Jul 14 20:26:06.368203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.368163 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" Jul 14 20:26:06.368203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.368177 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="635c4981-2e76-44c6-a805-1a1649d9dbf5" containerName="istio-proxy" Jul 14 20:26:06.372958 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.372939 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.377235 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.376497 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-a99cbd-openshift-default-dockercfg-7p52n\"" Jul 14 20:26:06.382777 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.382751 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c"] Jul 14 20:26:06.391680 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.391641 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:06.447624 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-socket\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447624 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447627 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-podinfo\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447651 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-certs\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447702 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-token\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447806 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-credential-socket\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447827 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-envoy\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447849 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-data\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.447907 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af84d5e5-7ecf-4c09-8573-7ba049148385-istiod-ca-cert\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.448264 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.447974 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzns\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-kube-api-access-gkzns\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.548645 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548600 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-certs\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.548645 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548639 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-token\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.548889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548701 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-credential-socket\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.548889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548718 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-envoy\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.548889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548740 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-data\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.548889 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548765 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af84d5e5-7ecf-4c09-8573-7ba049148385-istiod-ca-cert\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549135 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzns\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-kube-api-access-gkzns\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549135 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.548992 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-socket\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549135 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.549027 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-podinfo\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549296 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.549163 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-certs\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549296 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.549186 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-credential-socket\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549296 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.549259 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-data\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549464 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.549436 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-socket\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.549552 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.549533 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af84d5e5-7ecf-4c09-8573-7ba049148385-istiod-ca-cert\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.551370 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.551342 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-envoy\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.551474 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.551449 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-podinfo\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.557214 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.557194 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-token\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.557365 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.557348 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzns\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-kube-api-access-gkzns\") pod \"e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.559625 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.559601 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:06.688481 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.688444 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:06.827292 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.827268 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c"] Jul 14 20:26:06.829265 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:26:06.829236 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf84d5e5_7ecf_4c09_8573_7ba049148385.slice/crio-526f8d9928b9b93f434427c0a5e9b8e4a6766329e2ed3b2ec1a40ecc3bffa710 WatchSource:0}: Error finding container 526f8d9928b9b93f434427c0a5e9b8e4a6766329e2ed3b2ec1a40ecc3bffa710: Status 404 returned error can't find the container with id 526f8d9928b9b93f434427c0a5e9b8e4a6766329e2ed3b2ec1a40ecc3bffa710 Jul 14 20:26:06.831961 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.831928 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:06.832055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.831996 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:06.832055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:06.832028 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:07.317718 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:07.317682 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" event={"ID":"af84d5e5-7ecf-4c09-8573-7ba049148385","Type":"ContainerStarted","Data":"ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae"} Jul 14 20:26:07.318213 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:07.317732 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" event={"ID":"af84d5e5-7ecf-4c09-8573-7ba049148385","Type":"ContainerStarted","Data":"526f8d9928b9b93f434427c0a5e9b8e4a6766329e2ed3b2ec1a40ecc3bffa710"} Jul 14 20:26:07.340053 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:07.340005 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podStartSLOduration=1.339988461 podStartE2EDuration="1.339988461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:07.337723002 +0000 UTC m=+2372.786349443" watchObservedRunningTime="2026-07-14 20:26:07.339988461 +0000 UTC m=+2372.788614907" Jul 14 20:26:07.691189 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:07.691148 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:07.694826 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:07.694797 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:08.323571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:08.323543 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:08.324530 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:08.324513 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:09.241355 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.241299 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:09.299872 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.299840 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:26:09.341037 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.340992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:09.342537 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.342509 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7"] Jul 14 20:26:09.346133 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.346114 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.348307 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.348281 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a99cbd-serving-cert\"" Jul 14 20:26:09.358327 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.358301 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7"] Jul 14 20:26:09.410645 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.410614 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:09.491681 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.491571 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5daee2da-1035-472d-9c54-0cd26f11d505-maas-api-tls\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.491681 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.491620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daee2da-1035-472d-9c54-0cd26f11d505-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.491930 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.491708 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllx9\" (UniqueName: \"kubernetes.io/projected/5daee2da-1035-472d-9c54-0cd26f11d505-kube-api-access-dllx9\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.533891 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.533847 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:09.592573 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.592521 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5daee2da-1035-472d-9c54-0cd26f11d505-maas-api-tls\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.592573 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.592569 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daee2da-1035-472d-9c54-0cd26f11d505-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.592811 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.592595 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllx9\" (UniqueName: \"kubernetes.io/projected/5daee2da-1035-472d-9c54-0cd26f11d505-kube-api-access-dllx9\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.593247 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.593212 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daee2da-1035-472d-9c54-0cd26f11d505-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.595392 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.595367 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5daee2da-1035-472d-9c54-0cd26f11d505-maas-api-tls\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.600801 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.600775 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllx9\" (UniqueName: \"kubernetes.io/projected/5daee2da-1035-472d-9c54-0cd26f11d505-kube-api-access-dllx9\") pod \"maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.658803 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.658767 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:09.855051 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.855022 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7"] Jul 14 20:26:09.857264 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:26:09.857232 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daee2da_1035_472d_9c54_0cd26f11d505.slice/crio-0416591e031d8fcb811e54c2407acb15d76193bda804603e43b0465b1dd5e590 WatchSource:0}: Error finding container 0416591e031d8fcb811e54c2407acb15d76193bda804603e43b0465b1dd5e590: Status 404 returned error can't find the container with id 0416591e031d8fcb811e54c2407acb15d76193bda804603e43b0465b1dd5e590 Jul 14 20:26:09.938441 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:09.938403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:10.038507 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:10.038465 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:10.335813 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:10.335776 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" event={"ID":"5daee2da-1035-472d-9c54-0cd26f11d505","Type":"ContainerStarted","Data":"0416591e031d8fcb811e54c2407acb15d76193bda804603e43b0465b1dd5e590"} Jul 14 20:26:11.343278 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:11.343243 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" event={"ID":"5daee2da-1035-472d-9c54-0cd26f11d505","Type":"ContainerStarted","Data":"f9e2842d359c7e2030b0cf74037d93eed4b0cdf927a924a6fb2a04ea81ae4764"} Jul 14 20:26:11.343696 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:11.343317 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:11.362491 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:11.362444 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" podStartSLOduration=1.8149769999999998 podStartE2EDuration="2.362427113s" totalImagesPullingTime="547.450113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:11.361199607 +0000 UTC m=+2376.809826073" watchObservedRunningTime="2026-07-14 20:26:11.362427113 +0000 UTC m=+2376.811053614" Jul 14 20:26:12.670396 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:12.670358 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:13.760632 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:13.760599 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:15.271121 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.271052 2553 patch_prober.go:28] interesting pod/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 14 20:26:15.271775 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.271737 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 14 20:26:15.366847 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.366802 2553 generic.go:356] "Generic (PLEG): container finished" podID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerID="d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f" exitCode=0 Jul 14 20:26:15.367050 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.366880 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" event={"ID":"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b","Type":"ContainerDied","Data":"d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f"} Jul 14 20:26:15.367420 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.367397 2553 scope.go:117] "RemoveContainer" containerID="d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f" Jul 14 20:26:15.368921 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.368881 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:15.369006 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.368978 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:15.369091 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:15.369025 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:16.382168 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:16.381238 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" event={"ID":"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b","Type":"ContainerStarted","Data":"47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b"} Jul 14 20:26:16.578131 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:16.578092 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:16.578504 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:16.578470 2553 patch_prober.go:28] interesting pod/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 14 20:26:16.578724 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:16.578696 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 14 20:26:17.354763 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:17.354735 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:26:17.577813 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:17.577766 2553 patch_prober.go:28] interesting pod/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 14 20:26:17.578369 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:17.577837 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 14 20:26:18.578190 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:18.578149 2553 patch_prober.go:28] interesting pod/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 14 20:26:18.578706 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:18.578220 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 14 20:26:19.578335 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:19.578200 2553 patch_prober.go:28] interesting pod/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 14 20:26:19.578335 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:19.578279 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 14 20:26:20.578756 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:20.578288 2553 patch_prober.go:28] interesting pod/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 14 20:26:20.578756 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:20.578352 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 14 20:26:21.416203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.416169 2553 generic.go:356] "Generic (PLEG): container finished" podID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerID="ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae" exitCode=0 Jul 14 20:26:21.416421 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.416241 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" event={"ID":"af84d5e5-7ecf-4c09-8573-7ba049148385","Type":"ContainerDied","Data":"ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae"} Jul 14 20:26:21.416666 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.416652 2553 scope.go:117] "RemoveContainer" containerID="ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae" Jul 14 20:26:21.417632 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.417601 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:21.417831 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.417667 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:21.417831 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.417722 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:26:21.582110 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.582080 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:21.582571 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.582394 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:21.583027 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:21.583010 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:26:22.423856 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:22.423816 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" event={"ID":"af84d5e5-7ecf-4c09-8573-7ba049148385","Type":"ContainerStarted","Data":"406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88"} Jul 14 20:26:22.689320 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:22.689211 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:22.690233 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:22.690200 2553 patch_prober.go:28] interesting pod/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 20:26:22.690356 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:22.690280 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 20:26:23.324166 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:23.324119 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:23.689132 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:23.689097 2553 patch_prober.go:28] interesting pod/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 20:26:23.689827 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:23.689427 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 20:26:23.897168 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:23.897135 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:24.004848 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:24.004764 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:24.689657 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:24.689612 2553 patch_prober.go:28] interesting pod/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 20:26:24.689959 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:24.689924 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 20:26:25.689000 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:25.688956 2553 patch_prober.go:28] interesting pod/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 14 20:26:25.689348 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:25.689313 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 14 20:26:26.693111 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:26.693052 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:26.694234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:26.694210 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:26:27.236816 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:27.236778 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:30.181075 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:30.181024 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:30.880660 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:30.880631 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:30.981333 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:30.981299 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:26:35.140859 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:35.140828 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:26:35.148135 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:35.148112 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:26:53.119365 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:26:53.119330 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:27:50.488979 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:27:50.488912 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:27:50.550758 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:27:50.550723 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:29:05.294745 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:29:05.294708 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:29:07.257035 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:29:07.256986 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:29:15.832053 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:29:15.832013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:29:38.200381 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:29:38.200339 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:05.295374 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:05.295279 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:05.396754 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:05.396714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:24.009135 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:24.009090 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:24.106723 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:24.106685 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:27.568049 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:27.568007 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:28.502220 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:28.502187 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:30:35.136393 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:30:35.136350 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:31:11.428029 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:11.427993 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:31:24.821203 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:24.821140 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:31:24.923458 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:24.923423 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:31:31.893506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:31.893410 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:31:31.995565 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:31.995527 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:31:35.176037 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:35.175997 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:31:35.185589 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:31:35.185564 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddrpt_37d713aa-4f27-416c-b5cd-55a61898b273/ovn-acl-logging/0.log" Jul 14 20:32:13.189768 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:13.189734 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:14.202078 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:14.202017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:34.504234 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:34.504199 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:34.602101 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:34.602041 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:41.319188 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:41.319152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:44.288970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.288921 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp"] Jul 14 20:32:44.289835 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.289686 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" podUID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" containerName="maas-api" containerID="cri-o://788ab0bd01074b9b1a50c0071ffc08c2b141d88a1af8247c85e1afaaebc8f2c6" gracePeriod=30 Jul 14 20:32:44.294080 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.294039 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" podUID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.65:8443/health\": dial tcp 10.134.0.65:8443: connect: connection refused" Jul 14 20:32:44.313305 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.313277 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:44.488672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.488640 2553 generic.go:356] "Generic (PLEG): container finished" podID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" containerID="788ab0bd01074b9b1a50c0071ffc08c2b141d88a1af8247c85e1afaaebc8f2c6" exitCode=0 Jul 14 20:32:44.488818 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.488709 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" event={"ID":"834f3bf9-b411-46a7-8fa2-e0adff2d883f","Type":"ContainerDied","Data":"788ab0bd01074b9b1a50c0071ffc08c2b141d88a1af8247c85e1afaaebc8f2c6"} Jul 14 20:32:44.548013 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.547947 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:32:44.655970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.655930 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls\") pod \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " Jul 14 20:32:44.655970 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.655973 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nhlf\" (UniqueName: \"kubernetes.io/projected/834f3bf9-b411-46a7-8fa2-e0adff2d883f-kube-api-access-2nhlf\") pod \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " Jul 14 20:32:44.656216 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.656040 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/834f3bf9-b411-46a7-8fa2-e0adff2d883f-openshift-service-ca\") pod \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\" (UID: \"834f3bf9-b411-46a7-8fa2-e0adff2d883f\") " Jul 14 20:32:44.656486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.656457 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834f3bf9-b411-46a7-8fa2-e0adff2d883f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "834f3bf9-b411-46a7-8fa2-e0adff2d883f" (UID: "834f3bf9-b411-46a7-8fa2-e0adff2d883f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:32:44.658171 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.658140 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "834f3bf9-b411-46a7-8fa2-e0adff2d883f" (UID: "834f3bf9-b411-46a7-8fa2-e0adff2d883f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:32:44.658287 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.658202 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834f3bf9-b411-46a7-8fa2-e0adff2d883f-kube-api-access-2nhlf" (OuterVolumeSpecName: "kube-api-access-2nhlf") pod "834f3bf9-b411-46a7-8fa2-e0adff2d883f" (UID: "834f3bf9-b411-46a7-8fa2-e0adff2d883f"). InnerVolumeSpecName "kube-api-access-2nhlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:44.757736 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.757692 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/834f3bf9-b411-46a7-8fa2-e0adff2d883f-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:32:44.757736 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.757728 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/834f3bf9-b411-46a7-8fa2-e0adff2d883f-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:32:44.757736 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:44.757742 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nhlf\" (UniqueName: \"kubernetes.io/projected/834f3bf9-b411-46a7-8fa2-e0adff2d883f-kube-api-access-2nhlf\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:32:45.495288 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:45.495250 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" event={"ID":"834f3bf9-b411-46a7-8fa2-e0adff2d883f","Type":"ContainerDied","Data":"e1a7570b728b867cf661979d5096ae92885e7536f55d67f894eecdc260a7e106"} Jul 14 20:32:45.495781 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:45.495318 2553 scope.go:117] "RemoveContainer" containerID="788ab0bd01074b9b1a50c0071ffc08c2b141d88a1af8247c85e1afaaebc8f2c6" Jul 14 20:32:45.495781 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:45.495267 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp" Jul 14 20:32:45.518045 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:45.518003 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp"] Jul 14 20:32:45.532175 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:45.529923 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fad346-65bb84c99c-h94xp"] Jul 14 20:32:47.036882 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:47.036847 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" path="/var/lib/kubelet/pods/834f3bf9-b411-46a7-8fa2-e0adff2d883f/volumes" Jul 14 20:32:48.911454 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:48.911416 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:52.717502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:52.717471 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:55.227098 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:55.227039 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:55.304155 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:55.304119 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:59.613360 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:59.613269 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv"] Jul 14 20:32:59.613814 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:59.613542 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" containerID="cri-o://47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b" gracePeriod=30 Jul 14 20:32:59.638979 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:59.638943 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:32:59.867741 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:32:59.867654 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:00.736614 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:00.736578 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:00.839189 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:00.839152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:04.871372 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:04.871345 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:33:05.052797 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052770 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-data\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.052972 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052828 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-credential-socket\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.052972 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052850 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfgb2\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-kube-api-access-pfgb2\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.052972 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052871 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-socket\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.052972 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052889 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istiod-ca-cert\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.052972 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052928 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-podinfo\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.052972 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052956 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-envoy\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.053333 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.052977 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-certs\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.053333 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053004 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-token\") pod \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\" (UID: \"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b\") " Jul 14 20:33:05.053333 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053157 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:05.053486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053362 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.053486 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053408 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:05.053643 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053608 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:33:05.053714 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053633 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:05.053714 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.053649 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:05.055185 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.055162 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:05.055328 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.055246 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-kube-api-access-pfgb2" (OuterVolumeSpecName: "kube-api-access-pfgb2") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "kube-api-access-pfgb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:05.055541 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.055518 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:33:05.055625 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.055544 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" (UID: "0d2265f6-3d86-4c3f-ae57-9c60d9ae368b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:05.154368 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154325 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154368 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154356 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154368 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154367 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgb2\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-kube-api-access-pfgb2\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154368 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154376 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154386 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154394 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154401 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.154635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.154408 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:05.608252 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.608219 2553 generic.go:356] "Generic (PLEG): container finished" podID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerID="47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b" exitCode=0 Jul 14 20:33:05.608440 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.608269 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" Jul 14 20:33:05.608440 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.608301 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" event={"ID":"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b","Type":"ContainerDied","Data":"47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b"} Jul 14 20:33:05.608440 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.608356 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv" event={"ID":"0d2265f6-3d86-4c3f-ae57-9c60d9ae368b","Type":"ContainerDied","Data":"b68a18cd285848261b52bd3b07eacbdc348358a8a8d83c8855b28c362be9da18"} Jul 14 20:33:05.608440 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.608398 2553 scope.go:117] "RemoveContainer" containerID="47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b" Jul 14 20:33:05.617860 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.617840 2553 scope.go:117] "RemoveContainer" containerID="d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f" Jul 14 20:33:05.625776 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.625755 2553 scope.go:117] "RemoveContainer" containerID="47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b" Jul 14 20:33:05.626045 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:33:05.626021 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b\": container with ID starting with 47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b not found: ID does not exist" containerID="47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b" Jul 14 20:33:05.626139 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.626071 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b"} err="failed to get container status \"47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b\": rpc error: code = NotFound desc = could not find container \"47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b\": container with ID starting with 47e8c9998eaf59998cc6305060d7a856d40209ecab6bdf34e26d851c9afb390b not found: ID does not exist" Jul 14 20:33:05.626139 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.626099 2553 scope.go:117] "RemoveContainer" containerID="d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f" Jul 14 20:33:05.626355 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:33:05.626336 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f\": container with ID starting with d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f not found: ID does not exist" containerID="d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f" Jul 14 20:33:05.626396 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.626362 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f"} err="failed to get container status \"d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f\": rpc error: code = NotFound desc = could not find container \"d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f\": container with ID starting with d48bc2351a5e7d41091f3ae986d7126639f51e6503e4c9c791eaa4e108579f6f not found: ID does not exist" Jul 14 20:33:05.631540 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.631519 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv"] Jul 14 20:33:05.634639 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:05.634618 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fad346-openshift-default-8467cc97ff-wwwtv"] Jul 14 20:33:07.036217 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:07.036180 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" path="/var/lib/kubelet/pods/0d2265f6-3d86-4c3f-ae57-9c60d9ae368b/volumes" Jul 14 20:33:10.448752 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.448715 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7"] Jul 14 20:33:10.449244 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.448924 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" podUID="5daee2da-1035-472d-9c54-0cd26f11d505" containerName="maas-api" containerID="cri-o://f9e2842d359c7e2030b0cf74037d93eed4b0cdf927a924a6fb2a04ea81ae4764" gracePeriod=30 Jul 14 20:33:10.488246 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.488213 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:10.646591 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.646561 2553 generic.go:356] "Generic (PLEG): container finished" podID="5daee2da-1035-472d-9c54-0cd26f11d505" containerID="f9e2842d359c7e2030b0cf74037d93eed4b0cdf927a924a6fb2a04ea81ae4764" exitCode=0 Jul 14 20:33:10.646753 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.646632 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" event={"ID":"5daee2da-1035-472d-9c54-0cd26f11d505","Type":"ContainerDied","Data":"f9e2842d359c7e2030b0cf74037d93eed4b0cdf927a924a6fb2a04ea81ae4764"} Jul 14 20:33:10.688985 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.688961 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:33:10.702616 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.702561 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5daee2da-1035-472d-9c54-0cd26f11d505-maas-api-tls\") pod \"5daee2da-1035-472d-9c54-0cd26f11d505\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " Jul 14 20:33:10.702616 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.702605 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daee2da-1035-472d-9c54-0cd26f11d505-openshift-service-ca\") pod \"5daee2da-1035-472d-9c54-0cd26f11d505\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " Jul 14 20:33:10.702853 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.702691 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dllx9\" (UniqueName: \"kubernetes.io/projected/5daee2da-1035-472d-9c54-0cd26f11d505-kube-api-access-dllx9\") pod \"5daee2da-1035-472d-9c54-0cd26f11d505\" (UID: \"5daee2da-1035-472d-9c54-0cd26f11d505\") " Jul 14 20:33:10.703040 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.703017 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5daee2da-1035-472d-9c54-0cd26f11d505-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5daee2da-1035-472d-9c54-0cd26f11d505" (UID: "5daee2da-1035-472d-9c54-0cd26f11d505"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:33:10.704575 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.704556 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5daee2da-1035-472d-9c54-0cd26f11d505-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5daee2da-1035-472d-9c54-0cd26f11d505" (UID: "5daee2da-1035-472d-9c54-0cd26f11d505"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:33:10.704704 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.704686 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daee2da-1035-472d-9c54-0cd26f11d505-kube-api-access-dllx9" (OuterVolumeSpecName: "kube-api-access-dllx9") pod "5daee2da-1035-472d-9c54-0cd26f11d505" (UID: "5daee2da-1035-472d-9c54-0cd26f11d505"). InnerVolumeSpecName "kube-api-access-dllx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:10.803778 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.803741 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllx9\" (UniqueName: \"kubernetes.io/projected/5daee2da-1035-472d-9c54-0cd26f11d505-kube-api-access-dllx9\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:10.803778 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.803772 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5daee2da-1035-472d-9c54-0cd26f11d505-maas-api-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:10.803778 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:10.803782 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daee2da-1035-472d-9c54-0cd26f11d505-openshift-service-ca\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:11.653180 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:11.653150 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" Jul 14 20:33:11.653180 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:11.653174 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7" event={"ID":"5daee2da-1035-472d-9c54-0cd26f11d505","Type":"ContainerDied","Data":"0416591e031d8fcb811e54c2407acb15d76193bda804603e43b0465b1dd5e590"} Jul 14 20:33:11.653713 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:11.653237 2553 scope.go:117] "RemoveContainer" containerID="f9e2842d359c7e2030b0cf74037d93eed4b0cdf927a924a6fb2a04ea81ae4764" Jul 14 20:33:11.672528 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:11.672491 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7"] Jul 14 20:33:11.676411 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:11.676374 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a99cbd-6fcc9688f6-tzzp7"] Jul 14 20:33:13.036108 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:13.036047 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5daee2da-1035-472d-9c54-0cd26f11d505" path="/var/lib/kubelet/pods/5daee2da-1035-472d-9c54-0cd26f11d505/volumes" Jul 14 20:33:15.108206 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:15.108171 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:21.380955 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:21.380920 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:21.443205 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:21.443171 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:25.788498 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:25.788462 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c"] Jul 14 20:33:25.788964 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:25.788737 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" containerID="cri-o://406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88" gracePeriod=30 Jul 14 20:33:25.812324 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:25.812290 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:26.044162 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:26.044078 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:26.695548 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:26.695512 2553 patch_prober.go:28] interesting pod/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:33:26.695726 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:26.695580 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:33:31.044542 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.044516 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:33:31.189962 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.189927 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-envoy\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.189974 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-certs\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190013 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-token\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190040 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-credential-socket\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190055 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-data\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190110 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af84d5e5-7ecf-4c09-8573-7ba049148385-istiod-ca-cert\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190165 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-socket\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190204 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-podinfo\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190236 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkzns\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-kube-api-access-gkzns\") pod \"af84d5e5-7ecf-4c09-8573-7ba049148385\" (UID: \"af84d5e5-7ecf-4c09-8573-7ba049148385\") " Jul 14 20:33:31.190506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190390 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:31.190506 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190412 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:31.190723 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190517 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-credential-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.190723 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190537 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-certs\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.190723 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190568 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af84d5e5-7ecf-4c09-8573-7ba049148385-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:33:31.190723 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.190569 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:31.191206 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.191176 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-data" (OuterVolumeSpecName: "istio-data") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:31.192284 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.192251 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:31.192401 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.192366 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-token" (OuterVolumeSpecName: "istio-token") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:31.192470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.192415 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-kube-api-access-gkzns" (OuterVolumeSpecName: "kube-api-access-gkzns") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "kube-api-access-gkzns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:31.192533 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.192482 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af84d5e5-7ecf-4c09-8573-7ba049148385" (UID: "af84d5e5-7ecf-4c09-8573-7ba049148385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:33:31.291283 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291246 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-podinfo\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.291283 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291276 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzns\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-kube-api-access-gkzns\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.291283 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291288 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-envoy\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.291528 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291296 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-token\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.291528 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291304 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-istio-data\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.291528 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291311 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af84d5e5-7ecf-4c09-8573-7ba049148385-istiod-ca-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.291528 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.291319 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af84d5e5-7ecf-4c09-8573-7ba049148385-workload-socket\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\"" Jul 14 20:33:31.656840 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.656806 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-fc5448547-dfqhm_d28c1cba-dd0c-431c-9b00-3cae115cfc21/manager/0.log" Jul 14 20:33:31.759526 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.759496 2553 generic.go:356] "Generic (PLEG): container finished" podID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerID="406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88" exitCode=0 Jul 14 20:33:31.759716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.759545 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" Jul 14 20:33:31.759716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.759557 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" event={"ID":"af84d5e5-7ecf-4c09-8573-7ba049148385","Type":"ContainerDied","Data":"406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88"} Jul 14 20:33:31.759716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.759606 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c" event={"ID":"af84d5e5-7ecf-4c09-8573-7ba049148385","Type":"ContainerDied","Data":"526f8d9928b9b93f434427c0a5e9b8e4a6766329e2ed3b2ec1a40ecc3bffa710"} Jul 14 20:33:31.759716 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.759632 2553 scope.go:117] "RemoveContainer" containerID="406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88" Jul 14 20:33:31.769427 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.769409 2553 scope.go:117] "RemoveContainer" containerID="ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae" Jul 14 20:33:31.777883 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.777862 2553 scope.go:117] "RemoveContainer" containerID="406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88" Jul 14 20:33:31.778157 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:33:31.778136 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88\": container with ID starting with 406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88 not found: ID does not exist" containerID="406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88" Jul 14 20:33:31.778209 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.778165 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88"} err="failed to get container status \"406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88\": rpc error: code = NotFound desc = could not find container \"406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88\": container with ID starting with 406132139d5ab3356e72fb6e73aa9cdc045eadcc5bac25132866086808af5c88 not found: ID does not exist" Jul 14 20:33:31.778209 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.778183 2553 scope.go:117] "RemoveContainer" containerID="ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae" Jul 14 20:33:31.778407 ip-10-0-140-43 kubenswrapper[2553]: E0714 20:33:31.778389 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae\": container with ID starting with ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae not found: ID does not exist" containerID="ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae" Jul 14 20:33:31.778454 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.778413 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae"} err="failed to get container status \"ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae\": rpc error: code = NotFound desc = could not find container \"ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae\": container with ID starting with ab5cc81cd5a67300b27b49afd353152fb826c3e67b181cc5af8c4ef310b862ae not found: ID does not exist" Jul 14 20:33:31.785766 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.785736 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c"] Jul 14 20:33:31.792711 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:31.792688 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-a99cbd-openshift-default-d4d84cc8c-fmx2c"] Jul 14 20:33:32.278551 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:32.278521 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-pl58x_e8ef32cb-425c-4973-9e5b-18a520e9177e/copy-manifests/0.log" Jul 14 20:33:32.286114 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:32.286088 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-pl58x_e8ef32cb-425c-4973-9e5b-18a520e9177e/manager/0.log" Jul 14 20:33:32.676341 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:32.676312 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cf85b6845-tcngt_31eea769-ff9f-455b-a656-01392a686861/maas-api/0.log" Jul 14 20:33:32.804809 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:32.804783 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734320-sfkdw_566b7644-e4e0-4f42-bceb-b6921d984ca5/cleanup/0.log" Jul 14 20:33:33.036641 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:33.036563 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" path="/var/lib/kubelet/pods/af84d5e5-7ecf-4c09-8573-7ba049148385/volumes" Jul 14 20:33:33.984214 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:33.984187 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-f4a7e811-4shlt_4feb0822-1227-4a77-b917-bc6d057e5909/revoke-keys/0.log" Jul 14 20:33:34.550728 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:34.550705 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-676fd_87a76e58-b687-4c45-89ae-0a50dc6bc486/postgres/0.log" Jul 14 20:33:35.327311 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.327283 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484_73c6013e-2f06-43bf-ab99-e3e14b27ea62/util/0.log" Jul 14 20:33:35.343769 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.343745 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484_73c6013e-2f06-43bf-ab99-e3e14b27ea62/pull/0.log" Jul 14 20:33:35.361721 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.361692 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484_73c6013e-2f06-43bf-ab99-e3e14b27ea62/extract/0.log" Jul 14 20:33:35.477509 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.477478 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm_5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab/util/0.log" Jul 14 20:33:35.485104 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.485078 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm_5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab/pull/0.log" Jul 14 20:33:35.492158 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.492142 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm_5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab/extract/0.log" Jul 14 20:33:35.604457 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.604362 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk_1e3e7d92-e7c4-4cc3-9435-1328fc7db435/pull/0.log" Jul 14 20:33:35.611680 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.611659 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk_1e3e7d92-e7c4-4cc3-9435-1328fc7db435/extract/0.log" Jul 14 20:33:35.618502 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.618476 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk_1e3e7d92-e7c4-4cc3-9435-1328fc7db435/util/0.log" Jul 14 20:33:35.728488 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.728459 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm_a8b7f543-5c73-4adb-b711-13c87679d19c/util/0.log" Jul 14 20:33:35.735635 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.735596 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm_a8b7f543-5c73-4adb-b711-13c87679d19c/pull/0.log" Jul 14 20:33:35.742240 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:35.742214 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm_a8b7f543-5c73-4adb-b711-13c87679d19c/extract/0.log" Jul 14 20:33:37.230802 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:37.230770 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2jzhd_59228292-6033-4402-9935-3b486085902d/discovery/0.log" Jul 14 20:33:38.331826 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:38.331793 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-w2sl8_62fca802-ad0c-4c70-96c4-8949bdca550d/main/0.log" Jul 14 20:33:38.448971 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:38.448944 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2m54j_ce838c27-a756-4c04-9833-67272b283daf/main/0.log" Jul 14 20:33:42.946456 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946410 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v52j/must-gather-f66wt"] Jul 14 20:33:42.946941 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946924 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946944 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946956 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946965 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946976 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.946985 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947000 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" Jul 14 20:33:42.947008 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947009 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947034 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daee2da-1035-472d-9c54-0cd26f11d505" containerName="maas-api" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947043 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daee2da-1035-472d-9c54-0cd26f11d505" containerName="maas-api" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947076 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" containerName="maas-api" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947084 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" containerName="maas-api" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947179 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="834f3bf9-b411-46a7-8fa2-e0adff2d883f" containerName="maas-api" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947193 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daee2da-1035-472d-9c54-0cd26f11d505" containerName="maas-api" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947207 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947215 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947224 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2265f6-3d86-4c3f-ae57-9c60d9ae368b" containerName="istio-proxy" Jul 14 20:33:42.947465 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.947234 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="af84d5e5-7ecf-4c09-8573-7ba049148385" containerName="istio-proxy" Jul 14 20:33:42.952185 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.952165 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:42.954722 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.954703 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v52j\"/\"kube-root-ca.crt\"" Jul 14 20:33:42.955757 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.955734 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v52j\"/\"openshift-service-ca.crt\"" Jul 14 20:33:42.955850 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.955734 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7v52j\"/\"default-dockercfg-tbpgw\"" Jul 14 20:33:42.958289 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.958264 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/must-gather-f66wt"] Jul 14 20:33:42.964530 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:42.964505 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:43.080584 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.080545 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:43.092747 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.092702 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqj4\" (UniqueName: \"kubernetes.io/projected/7a41765f-b94a-4d5c-96b0-7421d350f362-kube-api-access-hpqj4\") pod \"must-gather-f66wt\" (UID: \"7a41765f-b94a-4d5c-96b0-7421d350f362\") " pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.092891 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.092768 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a41765f-b94a-4d5c-96b0-7421d350f362-must-gather-output\") pod \"must-gather-f66wt\" (UID: \"7a41765f-b94a-4d5c-96b0-7421d350f362\") " pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.193469 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.193423 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqj4\" (UniqueName: \"kubernetes.io/projected/7a41765f-b94a-4d5c-96b0-7421d350f362-kube-api-access-hpqj4\") pod \"must-gather-f66wt\" (UID: \"7a41765f-b94a-4d5c-96b0-7421d350f362\") " pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.193683 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.193493 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a41765f-b94a-4d5c-96b0-7421d350f362-must-gather-output\") pod \"must-gather-f66wt\" (UID: \"7a41765f-b94a-4d5c-96b0-7421d350f362\") " pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.193875 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.193853 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a41765f-b94a-4d5c-96b0-7421d350f362-must-gather-output\") pod \"must-gather-f66wt\" (UID: \"7a41765f-b94a-4d5c-96b0-7421d350f362\") " pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.202340 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.202268 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqj4\" (UniqueName: \"kubernetes.io/projected/7a41765f-b94a-4d5c-96b0-7421d350f362-kube-api-access-hpqj4\") pod \"must-gather-f66wt\" (UID: \"7a41765f-b94a-4d5c-96b0-7421d350f362\") " pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.262359 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.262325 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/must-gather-f66wt" Jul 14 20:33:43.397230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.397204 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/must-gather-f66wt"] Jul 14 20:33:43.399186 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:33:43.399155 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a41765f_b94a_4d5c_96b0_7421d350f362.slice/crio-a519da88b516e74781e6617c830dee551fce5dad8893d4c514050ba07deb5717 WatchSource:0}: Error finding container a519da88b516e74781e6617c830dee551fce5dad8893d4c514050ba07deb5717: Status 404 returned error can't find the container with id a519da88b516e74781e6617c830dee551fce5dad8893d4c514050ba07deb5717 Jul 14 20:33:43.401020 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.401003 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:33:43.821038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:43.821003 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/must-gather-f66wt" event={"ID":"7a41765f-b94a-4d5c-96b0-7421d350f362","Type":"ContainerStarted","Data":"a519da88b516e74781e6617c830dee551fce5dad8893d4c514050ba07deb5717"} Jul 14 20:33:45.837961 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:45.837922 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/must-gather-f66wt" event={"ID":"7a41765f-b94a-4d5c-96b0-7421d350f362","Type":"ContainerStarted","Data":"7c5687d6e994110a6916c4439f45baaa72306ba007c694b4866ee6d251591d14"} Jul 14 20:33:45.838409 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:45.837975 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/must-gather-f66wt" event={"ID":"7a41765f-b94a-4d5c-96b0-7421d350f362","Type":"ContainerStarted","Data":"bf63cf453a031f7deb835d06b217c0e409dc217ea56d2a8e28240dbdab7f6718"} Jul 14 20:33:45.856534 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:45.856473 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v52j/must-gather-f66wt" podStartSLOduration=2.460796819 podStartE2EDuration="3.856451704s" totalImagesPullingTime="1.395654885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:33:45.8546422 +0000 UTC m=+2831.303268646" watchObservedRunningTime="2026-07-14 20:33:45.856451704 +0000 UTC m=+2831.305078151" Jul 14 20:33:46.432149 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:46.432111 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-26pgr_437754cb-a459-4410-b75f-8ee2f10721ba/global-pull-secret-syncer/0.log" Jul 14 20:33:46.568519 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:46.568483 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" Jul 14 20:33:46.728496 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:46.728415 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-43.ec2.internal_452d5b1c76cd4f857f719491897be84c/haproxy/0.log" Jul 14 20:33:49.046852 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:49.046810 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:49.108326 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:49.108287 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2zjw5_c720d8d6-c35c-437f-bcbd-fee227e73d46/konnectivity-agent/0.log" limit=1024 Jul 14 20:33:51.380940 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.380905 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484_73c6013e-2f06-43bf-ab99-e3e14b27ea62/extract/0.log" Jul 14 20:33:51.416379 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.416350 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484_73c6013e-2f06-43bf-ab99-e3e14b27ea62/util/0.log" Jul 14 20:33:51.447175 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.447134 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75976484_73c6013e-2f06-43bf-ab99-e3e14b27ea62/pull/0.log" Jul 14 20:33:51.485407 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.485369 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm_5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab/extract/0.log" Jul 14 20:33:51.510414 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.510372 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm_5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab/util/0.log" Jul 14 20:33:51.535814 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.535738 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5xxm_5e21b8f7-d1b5-47c8-91d0-d3cf4b6146ab/pull/0.log" Jul 14 20:33:51.570327 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.570278 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk_1e3e7d92-e7c4-4cc3-9435-1328fc7db435/extract/0.log" Jul 14 20:33:51.596375 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.596330 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk_1e3e7d92-e7c4-4cc3-9435-1328fc7db435/util/0.log" Jul 14 20:33:51.622915 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.622883 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t6nnk_1e3e7d92-e7c4-4cc3-9435-1328fc7db435/pull/0.log" Jul 14 20:33:51.658803 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.658713 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm_a8b7f543-5c73-4adb-b711-13c87679d19c/extract/0.log" Jul 14 20:33:51.686628 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.686603 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm_a8b7f543-5c73-4adb-b711-13c87679d19c/util/0.log" Jul 14 20:33:51.709899 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:51.709870 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d7kgm_a8b7f543-5c73-4adb-b711-13c87679d19c/pull/0.log" Jul 14 20:33:54.341495 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.341462 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/alertmanager/0.log" Jul 14 20:33:54.367293 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.367263 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/config-reloader/0.log" Jul 14 20:33:54.391435 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.391379 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/kube-rbac-proxy-web/0.log" Jul 14 20:33:54.415040 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.415013 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/kube-rbac-proxy/0.log" Jul 14 20:33:54.438568 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.438532 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/kube-rbac-proxy-metric/0.log" Jul 14 20:33:54.462129 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.462102 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/prom-label-proxy/0.log" Jul 14 20:33:54.486038 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.486008 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dbbc5f1a-4050-4feb-abdf-264e33e700d3/init-config-reloader/0.log" Jul 14 20:33:54.638133 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.638052 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5688c564b9-qdd86_541fea56-0550-43d7-9a8e-5fbf794af13c/metrics-server/0.log" Jul 14 20:33:54.665833 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.665794 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-ldjhd_0e3e063e-0d08-4f83-a521-2aa4bfa27004/monitoring-plugin/0.log" Jul 14 20:33:54.784237 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.784204 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5lkw_837d41a4-fb76-4673-a1e6-67c5a9a3bca2/node-exporter/0.log" Jul 14 20:33:54.809575 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.809550 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5lkw_837d41a4-fb76-4673-a1e6-67c5a9a3bca2/kube-rbac-proxy/0.log" Jul 14 20:33:54.837553 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:54.837529 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5lkw_837d41a4-fb76-4673-a1e6-67c5a9a3bca2/init-textfile/0.log" Jul 14 20:33:55.013690 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.013614 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5lc9j_b1206c5d-c330-49e1-89ed-e5a98880fb3a/kube-rbac-proxy-main/0.log" Jul 14 20:33:55.037391 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.037361 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5lc9j_b1206c5d-c330-49e1-89ed-e5a98880fb3a/kube-rbac-proxy-self/0.log" Jul 14 20:33:55.064113 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.064087 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5lc9j_b1206c5d-c330-49e1-89ed-e5a98880fb3a/openshift-state-metrics/0.log" Jul 14 20:33:55.319230 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.319155 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-7zhwf_25e85cb9-250c-429b-bc86-3a8a0e7bfac6/prometheus-operator-admission-webhook/0.log" Jul 14 20:33:55.346671 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.346642 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d4f9f74d7-2qdfp_6f71e1e0-4c20-42a2-8fd0-a5f71ea399de/telemeter-client/0.log" Jul 14 20:33:55.383250 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.383223 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d4f9f74d7-2qdfp_6f71e1e0-4c20-42a2-8fd0-a5f71ea399de/reload/0.log" Jul 14 20:33:55.405176 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:55.405150 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d4f9f74d7-2qdfp_6f71e1e0-4c20-42a2-8fd0-a5f71ea399de/kube-rbac-proxy/0.log" Jul 14 20:33:56.450444 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.450408 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx"] Jul 14 20:33:56.458932 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.458891 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.469737 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.469704 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx"] Jul 14 20:33:56.529470 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.529420 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-sys\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.529749 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.529718 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-proc\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.529891 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.529874 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b964q\" (UniqueName: \"kubernetes.io/projected/d718b729-7bab-4442-8a91-87903f09a87c-kube-api-access-b964q\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.530024 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.530009 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-lib-modules\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.530175 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.530160 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-podres\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.631730 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.631682 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-sys\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.631933 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.631846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-proc\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.631933 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.631886 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b964q\" (UniqueName: \"kubernetes.io/projected/d718b729-7bab-4442-8a91-87903f09a87c-kube-api-access-b964q\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.631933 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.631802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-sys\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.632148 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.631917 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-lib-modules\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.632148 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.632001 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-podres\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.632148 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.632031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-lib-modules\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.632148 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.632133 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-podres\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.632394 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.632218 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d718b729-7bab-4442-8a91-87903f09a87c-proc\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.641785 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.641749 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b964q\" (UniqueName: \"kubernetes.io/projected/d718b729-7bab-4442-8a91-87903f09a87c-kube-api-access-b964q\") pod \"perf-node-gather-daemonset-w5gjx\" (UID: \"d718b729-7bab-4442-8a91-87903f09a87c\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.773668 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.773573 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:56.960516 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:56.960464 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx"] Jul 14 20:33:56.962956 ip-10-0-140-43 kubenswrapper[2553]: W0714 20:33:56.962908 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd718b729_7bab_4442_8a91_87903f09a87c.slice/crio-73b1aa5d2f7d1e0df61fd1b1be45a919971751f4cbaff5c8409720b82df1c6d8 WatchSource:0}: Error finding container 73b1aa5d2f7d1e0df61fd1b1be45a919971751f4cbaff5c8409720b82df1c6d8: Status 404 returned error can't find the container with id 73b1aa5d2f7d1e0df61fd1b1be45a919971751f4cbaff5c8409720b82df1c6d8 Jul 14 20:33:57.933466 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:57.933435 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" event={"ID":"d718b729-7bab-4442-8a91-87903f09a87c","Type":"ContainerStarted","Data":"16d417c859e37af390a7a62550f3f2b93a2fb323eb3694e39c80bb4e0d71cfd7"} Jul 14 20:33:57.933943 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:57.933486 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" event={"ID":"d718b729-7bab-4442-8a91-87903f09a87c","Type":"ContainerStarted","Data":"73b1aa5d2f7d1e0df61fd1b1be45a919971751f4cbaff5c8409720b82df1c6d8"} Jul 14 20:33:57.933943 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:57.933511 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:33:57.951319 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:57.951252 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" podStartSLOduration=1.951231617 podStartE2EDuration="1.951231617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:33:57.948403079 +0000 UTC m=+2843.397029522" watchObservedRunningTime="2026-07-14 20:33:57.951231617 +0000 UTC m=+2843.399858065" Jul 14 20:33:59.503055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:59.503026 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcsfp_995ad6e1-3b0a-462d-83b0-af9e5a1178fc/dns/0.log" Jul 14 20:33:59.527578 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:59.527544 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcsfp_995ad6e1-3b0a-462d-83b0-af9e5a1178fc/kube-rbac-proxy/0.log" Jul 14 20:33:59.578377 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:33:59.578346 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-njn4q_3cb36332-2f1b-4aee-a15b-951633f6e6cc/dns-node-resolver/0.log" Jul 14 20:34:00.158756 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:00.158715 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8f854fddd-dtm2f_a8249e15-4445-4b01-a11e-fe7a705b62b3/registry/0.log" Jul 14 20:34:00.181311 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:00.181277 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fxc4r_53011eae-e1ec-4f43-aee9-aa5ac3a52859/node-ca/0.log" Jul 14 20:34:01.355672 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:01.355646 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2jzhd_59228292-6033-4402-9935-3b486085902d/discovery/0.log" Jul 14 20:34:02.314703 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:02.314632 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-99jl6_f83805c3-112a-441e-a06b-dd01f9a1f2b1/serve-healthcheck-canary/0.log" Jul 14 20:34:02.859289 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:02.859259 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k9zs2_d35a5a8b-76ca-4120-9695-8e090145d939/kube-rbac-proxy/0.log" Jul 14 20:34:02.881341 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:02.881301 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k9zs2_d35a5a8b-76ca-4120-9695-8e090145d939/exporter/0.log" Jul 14 20:34:02.904269 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:02.904240 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k9zs2_d35a5a8b-76ca-4120-9695-8e090145d939/extractor/0.log" Jul 14 20:34:03.950554 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:03.950527 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-w5gjx" Jul 14 20:34:05.677599 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:05.677569 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-fc5448547-dfqhm_d28c1cba-dd0c-431c-9b00-3cae115cfc21/manager/0.log" Jul 14 20:34:05.990189 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:05.990161 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-pl58x_e8ef32cb-425c-4973-9e5b-18a520e9177e/manager/0.log" Jul 14 20:34:06.012695 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:06.012663 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-pl58x_e8ef32cb-425c-4973-9e5b-18a520e9177e/copy-manifests/0.log" Jul 14 20:34:06.809055 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:06.809015 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2jzhd_59228292-6033-4402-9935-3b486085902d/discovery/0.log" Jul 14 20:34:07.328726 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:07.328687 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dc78d48f4-skzmp_f9dd66bb-d2ee-4b0a-8124-79988bbfaf24/manager/0.log" Jul 14 20:34:07.357938 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:07.357904 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dwk65_ce986d47-fd2f-4a8f-a9de-0438f1b6ad44/openshift-lws-operator/0.log" Jul 14 20:34:08.391241 ip-10-0-140-43 kubenswrapper[2553]: I0714 20:34:08.391210 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2jzhd_59228292-6033-4402-9935-3b486085902d/discovery/0.log"