Jul 27 23:36:38.627869 ip-10-0-130-94 systemd[1]: Starting Kubernetes Kubelet... Jul 27 23:36:39.009875 ip-10-0-130-94 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 23:36:39.009875 ip-10-0-130-94 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 23:36:39.009875 ip-10-0-130-94 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 23:36:39.009875 ip-10-0-130-94 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 23:36:39.010534 ip-10-0-130-94 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 23:36:39.010534 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.009951 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 23:36:39.015178 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015161 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:36:39.015178 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015177 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015181 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015185 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015188 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015191 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015195 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015198 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015201 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015205 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015208 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015211 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015214 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015217 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015220 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015222 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015225 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015228 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015230 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015233 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015236 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:36:39.015244 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015238 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015246 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015249 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015253 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015256 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015259 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015264 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015268 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015271 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015274 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015276 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015279 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015282 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015285 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015288 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015291 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015293 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015296 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015298 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:36:39.015732 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015301 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015304 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015307 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015309 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015311 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015314 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015316 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015319 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015322 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015324 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015327 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015329 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015331 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015334 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015336 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015340 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015343 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015345 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015348 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015350 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:36:39.016188 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015352 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015355 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015358 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015360 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015363 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015365 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015367 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015371 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015374 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015376 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015379 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015382 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015384 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015387 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015389 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015392 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015394 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015397 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015399 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:36:39.016697 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015402 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015406 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015409 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015411 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015414 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015416 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015419 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015422 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015427 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015430 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015432 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015435 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015438 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015441 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015444 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015448 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015450 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015453 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015456 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.015459 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:36:39.017183 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016204 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016211 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016214 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016217 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016221 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016224 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016227 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016230 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016232 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016235 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016237 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016240 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016242 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016245 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016247 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016250 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016252 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016255 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016257 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:36:39.017677 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016261 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016266 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016270 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016272 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016275 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016278 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016281 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016283 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016286 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016289 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016291 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016294 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016296 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016299 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016301 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016304 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016307 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016309 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016312 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016315 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:36:39.018147 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016317 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016320 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016322 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016325 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016327 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016330 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016335 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016339 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016342 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016345 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016348 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016350 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016353 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016357 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016360 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016363 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016366 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016369 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016372 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:36:39.018675 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016374 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016377 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016379 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016382 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016384 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016387 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016389 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016392 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016394 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016397 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016400 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016402 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016407 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016410 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016413 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016416 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016419 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016422 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016424 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:36:39.019131 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016427 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016429 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016432 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016434 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016437 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016439 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016442 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016445 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016457 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016460 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016463 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016466 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016468 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016471 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016474 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016476 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016479 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016482 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016484 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016487 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:36:39.019589 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016489 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016492 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016889 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016894 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016896 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016899 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016903 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016905 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016908 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016911 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016913 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016916 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016918 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016921 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016923 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016926 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016928 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016931 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016933 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016936 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:36:39.020082 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016939 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016942 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016944 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016947 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016950 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016952 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016954 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016957 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016959 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016962 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016965 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016967 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016970 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016972 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016975 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016977 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016981 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016985 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016989 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:36:39.020580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016995 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.016998 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017000 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017003 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017006 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017009 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017011 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017015 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017017 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017020 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017022 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017025 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017030 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017032 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017035 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017038 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017040 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017043 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017045 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017048 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:36:39.021048 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017050 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017053 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017055 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017058 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017060 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017063 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017065 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017068 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017070 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017073 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017075 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017078 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017081 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017083 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017086 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017089 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017091 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017094 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017096 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017099 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:36:39.021543 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017101 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017104 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017106 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017109 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017114 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017117 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017120 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017123 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017126 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017128 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017131 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017135 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017138 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017140 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017142 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017145 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017148 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017150 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017153 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017155 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:36:39.022087 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017158 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017160 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017230 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017241 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017249 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017256 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017263 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017268 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017276 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017281 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017284 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017287 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017290 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017293 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017296 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017299 2555 flags.go:64] FLAG: --cgroup-root="" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017304 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017308 2555 flags.go:64] FLAG: --client-ca-file="" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017311 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017314 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017318 2555 flags.go:64] FLAG: --cluster-domain="" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017321 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017325 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017328 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 23:36:39.022584 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017331 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017335 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017338 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017341 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017344 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017347 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017350 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017353 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017356 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017359 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017363 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017366 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017369 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017372 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017375 2555 flags.go:64] FLAG: --enable-server="true" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017378 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017382 2555 flags.go:64] FLAG: --event-burst="100" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017386 2555 flags.go:64] FLAG: --event-qps="50" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017388 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017392 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017394 2555 flags.go:64] FLAG: --eviction-hard="" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017398 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017401 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017404 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017409 2555 flags.go:64] FLAG: --eviction-soft="" Jul 27 23:36:39.023169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017412 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017415 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017418 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017421 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017424 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017427 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017430 2555 flags.go:64] FLAG: --feature-gates="" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017434 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017437 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017440 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017443 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017446 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017449 2555 flags.go:64] FLAG: --help="false" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017453 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017456 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017459 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017461 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017465 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017468 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017471 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017474 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017477 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017480 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 23:36:39.023817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017483 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017486 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017489 2555 flags.go:64] FLAG: --kube-reserved="" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017492 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017495 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017498 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017501 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017503 2555 flags.go:64] FLAG: --lock-file="" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017508 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017510 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017513 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017519 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017521 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017525 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017528 2555 flags.go:64] FLAG: --logging-format="text" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017530 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017534 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017537 2555 flags.go:64] FLAG: --manifest-url="" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017539 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017543 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017546 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017550 2555 flags.go:64] FLAG: --max-pods="110" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017553 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017556 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017559 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 23:36:39.024424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017562 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017565 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017568 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017583 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017591 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017594 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017597 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017601 2555 flags.go:64] FLAG: --pod-cidr="" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017604 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017609 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017612 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017615 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017618 2555 flags.go:64] FLAG: --port="10250" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017621 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017624 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d0553c5b5da2de8" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017627 2555 flags.go:64] FLAG: --qos-reserved="" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017631 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017634 2555 flags.go:64] FLAG: --register-node="true" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017637 2555 flags.go:64] FLAG: --register-with-taints="" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017640 2555 flags.go:64] FLAG: --registry-burst="10" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017643 2555 flags.go:64] FLAG: --registry-qps="5" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017646 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017648 2555 flags.go:64] FLAG: --reserved-memory="" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017652 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017655 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 23:36:39.025114 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017658 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017661 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017664 2555 flags.go:64] FLAG: --runonce="false" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017666 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017669 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017672 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017675 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017678 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017681 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017684 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017687 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017690 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017693 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017696 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017700 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017703 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017707 2555 flags.go:64] FLAG: --system-cgroups="" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017710 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017715 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017718 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017720 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017724 2555 flags.go:64] FLAG: --tls-min-version="" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017727 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017731 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017734 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 23:36:39.025757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017736 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017739 2555 flags.go:64] FLAG: --v="2" Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017743 2555 flags.go:64] FLAG: --version="false" Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017747 2555 flags.go:64] FLAG: --vmodule="" Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017751 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.017754 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017878 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017881 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017885 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017888 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017892 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017895 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017898 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017901 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017903 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017907 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017910 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017912 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017915 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017918 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017921 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:36:39.026368 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017925 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017928 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017930 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017938 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017941 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017944 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017946 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017949 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017951 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017957 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017960 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017963 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017965 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017968 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017971 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017973 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017976 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017978 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017981 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017983 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:36:39.026916 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017986 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017988 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017991 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017993 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017996 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.017999 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018001 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018004 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018007 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018009 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018012 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018015 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018019 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018022 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018024 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018027 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018030 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018033 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018035 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018038 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:36:39.027420 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018040 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018044 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018046 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018049 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018052 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018054 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018057 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018059 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018062 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018064 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018067 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018069 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018072 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018076 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018079 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018082 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018085 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018088 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018091 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:36:39.027922 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018093 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018096 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018099 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018102 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018105 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018109 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018111 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018113 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018116 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018120 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018124 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018128 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018131 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018134 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018138 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018141 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018144 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018147 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018150 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:36:39.028593 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018153 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018155 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018158 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018161 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018164 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.018166 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.018758 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.025672 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.025698 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.025716 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.025722 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025790 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025795 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025799 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025802 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:36:39.029286 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025805 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025807 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025810 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025814 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025817 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025819 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025823 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025826 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025829 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025831 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025834 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025837 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025840 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025844 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025847 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025851 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025855 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025858 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025861 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:36:39.029749 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025864 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025867 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025870 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025872 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025875 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025878 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025887 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025890 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025892 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025895 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025897 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025900 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025903 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025906 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025908 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025911 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025913 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025916 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025919 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025921 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:36:39.030218 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025924 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025926 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025929 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025931 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025934 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025936 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025939 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025941 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025944 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025946 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025949 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025951 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025954 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025956 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025959 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025961 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025964 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025967 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025971 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:36:39.030728 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025980 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025984 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025987 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025990 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025993 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025996 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.025999 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026002 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026005 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026007 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026010 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026012 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026015 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026018 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026020 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026023 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026025 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026028 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026030 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:36:39.031192 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026033 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026036 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026038 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026040 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026043 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026046 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026048 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026051 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026053 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026056 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026058 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026061 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026064 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026067 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026075 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026078 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026081 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:36:39.031733 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026083 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.026088 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026252 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026257 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026261 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026264 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026267 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026270 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026272 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026275 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026278 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026280 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026283 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026286 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026288 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:36:39.032149 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026291 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026294 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026296 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026299 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026301 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026304 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026306 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026309 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026311 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026314 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026316 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026319 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026322 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026325 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026327 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026335 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026338 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026341 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026343 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:36:39.032532 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026347 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026350 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026354 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026358 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026361 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026364 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026367 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026370 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026373 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026376 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026379 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026381 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026384 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026387 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026389 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026392 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026394 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026397 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026399 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026402 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:36:39.033000 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026404 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026407 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026409 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026412 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026414 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026417 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026420 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026423 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026425 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026433 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026436 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026439 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026441 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026444 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026447 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026449 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026451 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026454 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026456 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026459 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:36:39.033466 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026461 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026464 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026466 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026470 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026473 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026476 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026479 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026482 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026485 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026487 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026490 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026492 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026495 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026497 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026499 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026502 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026505 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026508 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026510 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:36:39.033951 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026513 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026515 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026518 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026526 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026529 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026532 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026535 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:39.026537 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.026542 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.026703 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.029442 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.030903 2555 server.go:1019] "Starting client certificate rotation" Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.031000 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 23:36:39.034408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.031038 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 23:36:39.034775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.033349 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 23:36:39.035368 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.035351 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 23:36:39.047052 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.047036 2555 log.go:25] "Validated CRI v1 runtime API" Jul 27 23:36:39.048992 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.048980 2555 log.go:25] "Validated CRI v1 image API" Jul 27 23:36:39.049844 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.049829 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 23:36:39.053545 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.053523 2555 fs.go:135] Filesystem UUIDs: map[038227c7-2e0b-4b06-9760-5d6c36f66a39:/dev/nvme0n1p3 0ab900fa-48d3-4165-b4f9-202f4fb96ad4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 27 23:36:39.053614 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.053545 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 23:36:39.057293 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.057277 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 23:36:39.059090 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.058979 2555 manager.go:216] Machine: {Timestamp:2026-07-27 23:36:39.057463929 +0000 UTC m=+0.317594419 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101994 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb2cae99efb8d15548696b9239f6e SystemUUID:ec2bb2ca-e99e-fb8d-1554-8696b9239f6e BootID:ef253fa9-0819-4546-8e7a-8d82632ea991 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fb:23:2c:66:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fb:23:2c:66:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:aa:19:4e:f3:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 23:36:39.059090 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.059088 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 23:36:39.059191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.059166 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 23:36:39.062284 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.062261 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 23:36:39.062420 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.062287 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 23:36:39.063002 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.062992 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 23:36:39.063034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.063005 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 23:36:39.063034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.063012 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 23:36:39.063719 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.063708 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 23:36:39.064890 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.064880 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 23:36:39.067742 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067731 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 27 23:36:39.067869 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067859 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 23:36:39.067898 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067894 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 27 23:36:39.067925 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067903 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 23:36:39.067925 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067918 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 23:36:39.067981 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067926 2555 kubelet.go:398] "Adding apiserver pod source" Jul 27 23:36:39.067981 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.067941 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 23:36:39.068993 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.068962 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 23:36:39.069096 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.068999 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 23:36:39.072830 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.072813 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 23:36:39.074355 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.074339 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 23:36:39.074434 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.074378 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 23:36:39.075065 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075052 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 23:36:39.075148 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075072 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 23:36:39.075148 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075081 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 23:36:39.075148 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075089 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 23:36:39.075148 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075100 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 23:36:39.075148 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075108 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 23:36:39.075360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075207 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 23:36:39.075360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075217 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 23:36:39.075360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075227 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 23:36:39.075360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075235 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 23:36:39.075360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075248 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 23:36:39.075360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.075261 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 23:36:39.076742 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.076730 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 23:36:39.076809 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.076744 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 23:36:39.079812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.079797 2555 server.go:1268] "Started kubelet" Jul 27 23:36:39.079884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.079858 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 23:36:39.079985 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.079945 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 23:36:39.080051 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.080013 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 23:36:39.080201 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.080177 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 23:36:39.080313 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.080293 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 23:36:39.080385 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.080366 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 23:36:39.081422 ip-10-0-130-94 systemd[1]: Started Kubernetes Kubelet. Jul 27 23:36:39.082486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.081843 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 23:36:39.082962 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.082947 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 27 23:36:39.084038 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.084017 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qtxx5" Jul 27 23:36:39.084129 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.084039 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 23:36:39.088970 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.087970 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-94.ec2.internal.18c649c1d0cfbf90 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-94.ec2.internal,UID:ip-10-0-130-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-94.ec2.internal,},FirstTimestamp:2026-07-27 23:36:39.079772048 +0000 UTC m=+0.339902536,LastTimestamp:2026-07-27 23:36:39.079772048 +0000 UTC m=+0.339902536,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-94.ec2.internal,}" Jul 27 23:36:39.089760 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.089740 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 23:36:39.089819 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.089751 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 23:36:39.089979 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.089966 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.092010 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.090308 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 23:36:39.092010 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.090316 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 23:36:39.092010 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.090344 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 23:36:39.092010 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.090468 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 27 23:36:39.092010 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.090478 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 27 23:36:39.092010 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.091463 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qtxx5" Jul 27 23:36:39.092550 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.092531 2555 factory.go:55] Registering systemd factory Jul 27 23:36:39.092674 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.092662 2555 factory.go:223] Registration of the systemd container factory successfully Jul 27 23:36:39.093077 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.093053 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 23:36:39.093422 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.093396 2555 factory.go:153] Registering CRI-O factory Jul 27 23:36:39.093528 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.093424 2555 factory.go:223] Registration of the crio container factory successfully Jul 27 23:36:39.093528 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.093483 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 23:36:39.093528 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.093498 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 23:36:39.093528 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.093524 2555 factory.go:103] Registering Raw factory Jul 27 23:36:39.093729 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.093536 2555 manager.go:1196] Started watching for new ooms in manager Jul 27 23:36:39.093957 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.093944 2555 manager.go:349] Starting recovery of all containers Jul 27 23:36:39.105663 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.105645 2555 manager.go:354] Recovery completed Jul 27 23:36:39.109879 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.109867 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.111825 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.111807 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.111901 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.111839 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.111901 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.111853 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.112354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.112339 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 23:36:39.112354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.112348 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.112354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.112353 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 23:36:39.112496 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.112368 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 27 23:36:39.113594 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.113556 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.113674 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.113601 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.113674 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.113615 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.113766 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.113683 2555 policy_none.go:49] "None policy: Start" Jul 27 23:36:39.113766 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.113701 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 23:36:39.113766 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.113715 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 23:36:39.114954 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.114941 2555 policy_none.go:47] "Start" Jul 27 23:36:39.140251 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140233 2555 manager.go:337] "Starting Device Plugin manager" Jul 27 23:36:39.140363 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.140274 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 23:36:39.140363 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140291 2555 server.go:87] "Starting device plugin registration server" Jul 27 23:36:39.140554 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140543 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 23:36:39.140619 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140556 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 23:36:39.140676 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140637 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 23:36:39.140728 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140714 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 23:36:39.140728 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.140723 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 23:36:39.141545 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.141383 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 23:36:39.141646 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.141587 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.221505 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.221470 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 23:36:39.222711 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.222599 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 23:36:39.222711 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.222634 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 23:36:39.222711 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.222662 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 23:36:39.222850 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.222710 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 23:36:39.224872 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.224850 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:36:39.241635 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.241610 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.242524 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.242507 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.242623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.242541 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.242623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.242555 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.242623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.242601 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.248233 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.248219 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.248282 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.248244 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-94.ec2.internal\": node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.260738 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.260655 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.323035 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.323000 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal"] Jul 27 23:36:39.323244 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.323071 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.324662 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.324636 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.324756 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.324671 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.324756 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.324685 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.326076 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326061 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.326189 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326173 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.326237 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326207 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.326775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326761 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.326853 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326793 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.326853 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326807 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.326853 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326761 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.326987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326857 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.326987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.326871 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.328485 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.328467 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.328565 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.328490 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:36:39.329142 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.329127 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:36:39.329223 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.329157 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:36:39.329223 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.329171 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:36:39.352194 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.352176 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-94.ec2.internal\" not found" node="ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.356646 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.356628 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-94.ec2.internal\" not found" node="ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.361135 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.361120 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.392076 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.392048 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6d5157fea5c68626415a727705ed8a3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-94.ec2.internal\" (UID: \"a6d5157fea5c68626415a727705ed8a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.392181 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.392082 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/019df062654c9f634ff71e54cb9de65c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"019df062654c9f634ff71e54cb9de65c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.392181 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.392099 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/019df062654c9f634ff71e54cb9de65c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"019df062654c9f634ff71e54cb9de65c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.462266 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.462232 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.492635 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.492595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6d5157fea5c68626415a727705ed8a3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-94.ec2.internal\" (UID: \"a6d5157fea5c68626415a727705ed8a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.492635 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.492639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/019df062654c9f634ff71e54cb9de65c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"019df062654c9f634ff71e54cb9de65c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.492739 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.492657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/019df062654c9f634ff71e54cb9de65c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"019df062654c9f634ff71e54cb9de65c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.492739 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.492700 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/019df062654c9f634ff71e54cb9de65c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"019df062654c9f634ff71e54cb9de65c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.492739 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.492711 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6d5157fea5c68626415a727705ed8a3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-94.ec2.internal\" (UID: \"a6d5157fea5c68626415a727705ed8a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.492739 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.492711 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/019df062654c9f634ff71e54cb9de65c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"019df062654c9f634ff71e54cb9de65c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.562930 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.562853 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.654558 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.654529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.659656 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.659638 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jul 27 23:36:39.663607 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.663588 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.764104 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.764062 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.864686 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.864616 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:39.901526 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:39.901503 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:36:39.965475 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:39.965441 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:41.366856 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.366832 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 23:36:41.367238 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.366974 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 23:36:41.367238 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.367005 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 23:36:41.402149 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:41.402121 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:41.430873 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.430823 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 23:31:40 +0000 UTC" deadline="2028-03-13 12:21:35.955426664 +0000 UTC" Jul 27 23:36:41.430873 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.430867 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14268h44m54.524562449s" Jul 27 23:36:41.438452 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.438431 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:36:41.485065 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:41.485027 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d5157fea5c68626415a727705ed8a3.slice/crio-ed7025fe72890eaa4c8b84f40fa4748914f8e004112393cfb1555c98a9f35032 WatchSource:0}: Error finding container ed7025fe72890eaa4c8b84f40fa4748914f8e004112393cfb1555c98a9f35032: Status 404 returned error can't find the container with id ed7025fe72890eaa4c8b84f40fa4748914f8e004112393cfb1555c98a9f35032 Jul 27 23:36:41.485270 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:41.485252 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019df062654c9f634ff71e54cb9de65c.slice/crio-6d6eda0fee2fef144d877d5ca367343f190e7d265d38928c4c04f825ee9a0042 WatchSource:0}: Error finding container 6d6eda0fee2fef144d877d5ca367343f190e7d265d38928c4c04f825ee9a0042: Status 404 returned error can't find the container with id 6d6eda0fee2fef144d877d5ca367343f190e7d265d38928c4c04f825ee9a0042 Jul 27 23:36:41.489699 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.489681 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 23:36:41.502630 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:41.502608 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:41.562325 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.562270 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"019df062654c9f634ff71e54cb9de65c","Type":"ContainerStarted","Data":"6d6eda0fee2fef144d877d5ca367343f190e7d265d38928c4c04f825ee9a0042"} Jul 27 23:36:41.563274 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.563251 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" event={"ID":"a6d5157fea5c68626415a727705ed8a3","Type":"ContainerStarted","Data":"ed7025fe72890eaa4c8b84f40fa4748914f8e004112393cfb1555c98a9f35032"} Jul 27 23:36:41.603403 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:41.603375 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:41.703888 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:41.703859 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jul 27 23:36:41.727993 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.727941 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:36:41.827109 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.827081 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jul 27 23:36:41.837125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.837100 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 23:36:41.838305 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.838288 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jul 27 23:36:41.843764 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:41.843750 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 23:36:42.371073 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.371044 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:36:42.404873 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.404789 2555 apiserver.go:52] "Watching apiserver" Jul 27 23:36:42.411249 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.411220 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 23:36:42.411806 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.411774 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-k9p72","openshift-ovn-kubernetes/ovnkube-node-7xjmj","kube-system/global-pull-secret-syncer-gqttn","kube-system/konnectivity-agent-cmgpx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk","openshift-dns/node-resolver-n688k","openshift-multus/multus-additional-cni-plugins-pm6m5","openshift-multus/network-metrics-daemon-ppxkd","openshift-network-diagnostics/network-check-target-89dpp","openshift-network-operator/iptables-alerter-jvzwz","kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rx964","openshift-image-registry/node-ca-s7jtw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal"] Jul 27 23:36:42.413550 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.413529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.415055 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.415037 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.416129 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.416106 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 23:36:42.417909 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.416950 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.417909 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.417029 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:42.417909 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.417079 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.417909 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.417415 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 23:36:42.417909 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.417673 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 23:36:42.418178 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.418154 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.418178 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.418161 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbqzt\"" Jul 27 23:36:42.418419 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.418160 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8d749\"" Jul 27 23:36:42.419808 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.418749 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.419808 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.418964 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 23:36:42.419808 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.419158 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.420869 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.420309 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.422163 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.422142 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.422163 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.422154 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 23:36:42.422386 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.422287 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 23:36:42.422386 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.422331 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c76qs\"" Jul 27 23:36:42.424263 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.423807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.424752 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.424731 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 23:36:42.424855 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.424837 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-47kzq\"" Jul 27 23:36:42.424984 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.424969 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.425132 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.425115 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:42.425200 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.425177 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:42.425323 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.425309 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.425755 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.425738 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.425755 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.425754 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.425899 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.425799 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74dl6\"" Jul 27 23:36:42.425899 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.425888 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 23:36:42.426375 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.426265 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.427956 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.427938 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.428057 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.428040 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 23:36:42.428268 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.428253 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vt7nb\"" Jul 27 23:36:42.429325 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.429305 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.430117 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430100 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 23:36:42.430198 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430142 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.430256 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430206 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rf8cr\"" Jul 27 23:36:42.430489 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430441 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.430749 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430693 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 23:36:42.430875 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430847 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:42.430943 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430714 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 23:36:42.430943 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.430921 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:42.430943 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.430795 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 23:36:42.431341 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.431323 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.431862 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.431598 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.431862 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.431606 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 23:36:42.432006 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.431921 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z5ndr\"" Jul 27 23:36:42.432251 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.432231 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.434288 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.434269 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:36:42.434519 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.434484 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 23:36:42.434962 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.434940 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 23:36:42.435055 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.434972 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nt2r8\"" Jul 27 23:36:42.439393 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439371 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-env-overrides\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.439485 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-os-release\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.439485 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.439485 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439464 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-registration-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.439630 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439488 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-cni-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.439630 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439537 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45760ac0-5980-4697-aaa1-c7945a19b317-cni-binary-copy\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.439630 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439559 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d50a704-4e89-4af2-ad0f-d7ad884eaa1e-agent-certs\") pod \"konnectivity-agent-cmgpx\" (UID: \"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e\") " pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.439630 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439580 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-systemd-units\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.439630 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439604 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.439630 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439628 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-system-cni-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-ovnkube-script-lib\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439674 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cni-binary-copy\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439697 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-sys-fs\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25d14687-0f5b-47e0-9d9c-20c33d00e49e-hosts-file\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439751 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-cni-multus\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439780 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a704-4e89-4af2-ad0f-d7ad884eaa1e-konnectivity-ca\") pod \"konnectivity-agent-cmgpx\" (UID: \"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e\") " pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439803 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d14687-0f5b-47e0-9d9c-20c33d00e49e-tmp-dir\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439831 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxkw\" (UniqueName: \"kubernetes.io/projected/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-kube-api-access-fsxkw\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-node-log\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439878 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-socket-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.439912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439901 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-conf-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-ovnkube-config\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zxm\" (UniqueName: \"kubernetes.io/projected/18bb38c8-b12b-4248-af07-d67fc322c56a-kube-api-access-m2zxm\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.439985 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-run-netns\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440006 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-var-lib-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440041 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440065 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-serviceca\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440088 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-etc-selinux\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440110 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-multus-certs\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440132 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-cni-bin\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440156 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-host\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440207 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-kubelet\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440240 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-kubelet\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440267 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-system-cni-dir\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440291 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-dbus\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.440418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440314 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpg9v\" (UniqueName: \"kubernetes.io/projected/25d14687-0f5b-47e0-9d9c-20c33d00e49e-kube-api-access-gpg9v\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-os-release\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-systemd\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440391 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440416 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-kubelet-config\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440444 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-cni-bin\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440470 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqh5\" (UniqueName: \"kubernetes.io/projected/45760ac0-5980-4697-aaa1-c7945a19b317-kube-api-access-lbqh5\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440509 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cnibin\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440534 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440568 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-hostroot\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440591 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-k8s-cni-cncf-io\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440618 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-etc-kubernetes\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440643 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-etc-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440676 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-cnibin\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440700 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-slash\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440724 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18bb38c8-b12b-4248-af07-d67fc322c56a-ovn-node-metrics-cert\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.441447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440750 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-device-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.440815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnj6\" (UniqueName: \"kubernetes.io/projected/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-kube-api-access-zqnj6\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-socket-dir-parent\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441382 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-ovn\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441404 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-log-socket\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441427 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441447 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-cni-netd\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441471 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441512 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-netns\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45760ac0-5980-4697-aaa1-c7945a19b317-multus-daemon-config\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.442230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.441581 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vm2j\" (UniqueName: \"kubernetes.io/projected/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-kube-api-access-8vm2j\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.459485 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.459461 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d5fbz" Jul 27 23:36:42.467665 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.467645 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d5fbz" Jul 27 23:36:42.527884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.527853 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 23:36:42.541956 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.541920 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:42.541956 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.541966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-kubelet\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.541996 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-kubelet\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542019 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-system-cni-dir\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542043 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-dbus\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542070 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70962f98-5b88-4104-96d6-7fb957944574-tmp\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542080 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-kubelet\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542087 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-system-cni-dir\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542095 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpg9v\" (UniqueName: \"kubernetes.io/projected/25d14687-0f5b-47e0-9d9c-20c33d00e49e-kube-api-access-gpg9v\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542133 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-kubelet\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-os-release\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-systemd\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.542190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542194 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542201 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-os-release\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542218 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-kubelet-config\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542241 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-systemd\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542244 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-sys\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542282 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-cni-bin\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542290 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-kubelet-config\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542315 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqh5\" (UniqueName: \"kubernetes.io/projected/45760ac0-5980-4697-aaa1-c7945a19b317-kube-api-access-lbqh5\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542321 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-cni-bin\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542353 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cnibin\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542350 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-dbus\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542392 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cnibin\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542431 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542460 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-hostroot\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-k8s-cni-cncf-io\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542551 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-etc-kubernetes\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.542817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542554 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-hostroot\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.542566 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542573 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-etc-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542598 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysconfig\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542604 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-k8s-cni-cncf-io\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542616 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-etc-kubernetes\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542622 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-cnibin\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542636 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-etc-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.542645 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:43.04262401 +0000 UTC m=+2.966832869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542667 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-cnibin\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542686 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-slash\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542714 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18bb38c8-b12b-4248-af07-d67fc322c56a-ovn-node-metrics-cert\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542741 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d212a78-26c3-46de-8dfe-f299488fdf2d-iptables-alerter-script\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542748 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-slash\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542768 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5zp\" (UniqueName: \"kubernetes.io/projected/0d212a78-26c3-46de-8dfe-f299488fdf2d-kube-api-access-6m5zp\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysctl-d\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542835 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-host\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.543654 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-device-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542903 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnj6\" (UniqueName: \"kubernetes.io/projected/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-kube-api-access-zqnj6\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542929 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-socket-dir-parent\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542964 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-device-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.542976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-ovn\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543013 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-socket-dir-parent\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-log-socket\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543059 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-ovn\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543059 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-run-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543050 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543062 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543104 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-log-socket\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543102 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-cni-netd\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543132 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543135 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543156 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.544454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543167 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-netns\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543190 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45760ac0-5980-4697-aaa1-c7945a19b317-multus-daemon-config\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543192 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-cni-netd\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543214 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-netns\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543275 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vm2j\" (UniqueName: \"kubernetes.io/projected/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-kube-api-access-8vm2j\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d212a78-26c3-46de-8dfe-f299488fdf2d-host-slash\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543311 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-run\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543331 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70962f98-5b88-4104-96d6-7fb957944574-etc-tuned\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543364 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-env-overrides\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-os-release\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543402 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543426 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-registration-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543443 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-cni-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543456 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45760ac0-5980-4697-aaa1-c7945a19b317-cni-binary-copy\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543471 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-os-release\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543479 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d50a704-4e89-4af2-ad0f-d7ad884eaa1e-agent-certs\") pod \"konnectivity-agent-cmgpx\" (UID: \"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e\") " pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543535 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-systemd-units\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.545286 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543537 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-registration-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543561 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543576 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-cni-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543586 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-system-cni-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543593 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543620 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-ovnkube-script-lib\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543644 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-system-cni-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543682 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-systemd-units\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543776 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45760ac0-5980-4697-aaa1-c7945a19b317-multus-daemon-config\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543846 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cni-binary-copy\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543872 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-sys-fs\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543893 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25d14687-0f5b-47e0-9d9c-20c33d00e49e-hosts-file\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543917 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-cni-multus\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543934 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-env-overrides\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543942 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a704-4e89-4af2-ad0f-d7ad884eaa1e-konnectivity-ca\") pod \"konnectivity-agent-cmgpx\" (UID: \"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e\") " pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543971 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-lib-modules\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543999 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqj2\" (UniqueName: \"kubernetes.io/projected/3e25d840-3aaa-417d-9ede-4a6e9f79d730-kube-api-access-xjqj2\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:42.545983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544005 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25d14687-0f5b-47e0-9d9c-20c33d00e49e-hosts-file\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544023 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-var-lib-cni-multus\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544025 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d14687-0f5b-47e0-9d9c-20c33d00e49e-tmp-dir\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544066 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxkw\" (UniqueName: \"kubernetes.io/projected/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-kube-api-access-fsxkw\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-kubernetes\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-node-log\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544130 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-var-lib-kubelet\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544155 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-socket-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544161 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544173 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-conf-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544161 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-ovnkube-script-lib\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544193 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-systemd\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544203 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-node-log\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544223 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5kt\" (UniqueName: \"kubernetes.io/projected/70962f98-5b88-4104-96d6-7fb957944574-kube-api-access-nh5kt\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544252 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-ovnkube-config\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544276 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d14687-0f5b-47e0-9d9c-20c33d00e49e-tmp-dir\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544300 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-socket-dir\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.546637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544346 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-multus-conf-dir\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.543969 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45760ac0-5980-4697-aaa1-c7945a19b317-cni-binary-copy\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-sys-fs\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544278 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zxm\" (UniqueName: \"kubernetes.io/projected/18bb38c8-b12b-4248-af07-d67fc322c56a-kube-api-access-m2zxm\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-run-netns\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544520 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-cni-binary-copy\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-var-lib-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544576 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-var-lib-openvswitch\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544581 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-run-netns\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-serviceca\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-modprobe-d\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-etc-selinux\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-multus-certs\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544754 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-cni-bin\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-host\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544805 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysctl-conf\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.547367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544827 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18bb38c8-b12b-4248-af07-d67fc322c56a-ovnkube-config\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544884 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a704-4e89-4af2-ad0f-d7ad884eaa1e-konnectivity-ca\") pod \"konnectivity-agent-cmgpx\" (UID: \"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e\") " pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.544931 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18bb38c8-b12b-4248-af07-d67fc322c56a-host-cni-bin\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.545006 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-etc-selinux\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.545041 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-host\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.545096 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45760ac0-5980-4697-aaa1-c7945a19b317-host-run-multus-certs\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.545246 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-serviceca\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.545371 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.547064 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18bb38c8-b12b-4248-af07-d67fc322c56a-ovn-node-metrics-cert\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.547168 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d50a704-4e89-4af2-ad0f-d7ad884eaa1e-agent-certs\") pod \"konnectivity-agent-cmgpx\" (UID: \"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e\") " pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.547254 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.547274 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.547286 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:42.548037 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.547345 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:43.047327779 +0000 UTC m=+2.971536652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:42.549425 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.549381 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpg9v\" (UniqueName: \"kubernetes.io/projected/25d14687-0f5b-47e0-9d9c-20c33d00e49e-kube-api-access-gpg9v\") pod \"node-resolver-n688k\" (UID: \"25d14687-0f5b-47e0-9d9c-20c33d00e49e\") " pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.550552 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.550529 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnj6\" (UniqueName: \"kubernetes.io/projected/bf5b9d0b-3470-4f33-bf0b-30c1cd569c42-kube-api-access-zqnj6\") pod \"aws-ebs-csi-driver-node-ff5gk\" (UID: \"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.550789 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.550772 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqh5\" (UniqueName: \"kubernetes.io/projected/45760ac0-5980-4697-aaa1-c7945a19b317-kube-api-access-lbqh5\") pod \"multus-k9p72\" (UID: \"45760ac0-5980-4697-aaa1-c7945a19b317\") " pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.553350 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.553328 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vm2j\" (UniqueName: \"kubernetes.io/projected/cc3eb81d-5605-4d0a-a7ec-6e952d769f35-kube-api-access-8vm2j\") pod \"node-ca-s7jtw\" (UID: \"cc3eb81d-5605-4d0a-a7ec-6e952d769f35\") " pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.553558 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.553538 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxkw\" (UniqueName: \"kubernetes.io/projected/3a3a0fc3-cab7-48ac-9e8d-e04de7418400-kube-api-access-fsxkw\") pod \"multus-additional-cni-plugins-pm6m5\" (UID: \"3a3a0fc3-cab7-48ac-9e8d-e04de7418400\") " pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.554372 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.554352 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zxm\" (UniqueName: \"kubernetes.io/projected/18bb38c8-b12b-4248-af07-d67fc322c56a-kube-api-access-m2zxm\") pod \"ovnkube-node-7xjmj\" (UID: \"18bb38c8-b12b-4248-af07-d67fc322c56a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.645898 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645817 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-kubernetes\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.645898 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645857 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-var-lib-kubelet\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.645898 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-systemd\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645950 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-systemd\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5kt\" (UniqueName: \"kubernetes.io/projected/70962f98-5b88-4104-96d6-7fb957944574-kube-api-access-nh5kt\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645950 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-var-lib-kubelet\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.645964 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-kubernetes\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646009 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-modprobe-d\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysctl-conf\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646095 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70962f98-5b88-4104-96d6-7fb957944574-tmp\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646120 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-sys\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646151 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:42.646165 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646157 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-modprobe-d\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646173 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysconfig\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d212a78-26c3-46de-8dfe-f299488fdf2d-iptables-alerter-script\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646213 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-sys\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646226 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5zp\" (UniqueName: \"kubernetes.io/projected/0d212a78-26c3-46de-8dfe-f299488fdf2d-kube-api-access-6m5zp\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646249 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysctl-d\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646260 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysconfig\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646216 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysctl-conf\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646271 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-host\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.646309 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-host\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:42.646374 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:43.146355708 +0000 UTC m=+3.070564587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646464 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d212a78-26c3-46de-8dfe-f299488fdf2d-host-slash\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646515 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d212a78-26c3-46de-8dfe-f299488fdf2d-host-slash\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646518 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-run\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70962f98-5b88-4104-96d6-7fb957944574-etc-tuned\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646468 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-etc-sysctl-d\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.646726 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-lib-modules\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.647421 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646602 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-run\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.647421 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646616 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqj2\" (UniqueName: \"kubernetes.io/projected/3e25d840-3aaa-417d-9ede-4a6e9f79d730-kube-api-access-xjqj2\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:42.647421 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646725 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70962f98-5b88-4104-96d6-7fb957944574-lib-modules\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.647421 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.646875 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d212a78-26c3-46de-8dfe-f299488fdf2d-iptables-alerter-script\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.648908 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.648886 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70962f98-5b88-4104-96d6-7fb957944574-tmp\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.649007 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.648940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70962f98-5b88-4104-96d6-7fb957944574-etc-tuned\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.654215 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.654159 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5kt\" (UniqueName: \"kubernetes.io/projected/70962f98-5b88-4104-96d6-7fb957944574-kube-api-access-nh5kt\") pod \"tuned-rx964\" (UID: \"70962f98-5b88-4104-96d6-7fb957944574\") " pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:42.654337 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.654317 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5zp\" (UniqueName: \"kubernetes.io/projected/0d212a78-26c3-46de-8dfe-f299488fdf2d-kube-api-access-6m5zp\") pod \"iptables-alerter-jvzwz\" (UID: \"0d212a78-26c3-46de-8dfe-f299488fdf2d\") " pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.654394 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.654354 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqj2\" (UniqueName: \"kubernetes.io/projected/3e25d840-3aaa-417d-9ede-4a6e9f79d730-kube-api-access-xjqj2\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:42.728138 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.728109 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" Jul 27 23:36:42.737838 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.737812 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s7jtw" Jul 27 23:36:42.745513 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.745480 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:36:42.753045 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.753025 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" Jul 27 23:36:42.761465 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.761445 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n688k" Jul 27 23:36:42.768120 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.768099 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k9p72" Jul 27 23:36:42.776718 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.776700 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:36:42.783339 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.783322 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvzwz" Jul 27 23:36:42.788825 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:42.788807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rx964" Jul 27 23:36:43.049459 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.049384 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:43.049459 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.049426 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:43.049643 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.049542 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:43.049643 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.049591 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:36:43.049643 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.049619 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:36:43.049643 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.049632 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:43.049756 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.049596 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:44.049582443 +0000 UTC m=+3.973791300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:43.049756 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.049704 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:44.049686457 +0000 UTC m=+3.973895326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:43.150759 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.150721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:43.150919 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.150866 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:43.150966 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:43.150942 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:44.150919707 +0000 UTC m=+4.075128565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:43.246003 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.245969 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:36:43.357258 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.357222 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d50a704_4e89_4af2_ad0f_d7ad884eaa1e.slice/crio-29415ebdcf5b2d5e71a40b88941e6b7c292aafaf216d1805f311a8ef2fef463d WatchSource:0}: Error finding container 29415ebdcf5b2d5e71a40b88941e6b7c292aafaf216d1805f311a8ef2fef463d: Status 404 returned error can't find the container with id 29415ebdcf5b2d5e71a40b88941e6b7c292aafaf216d1805f311a8ef2fef463d Jul 27 23:36:43.361157 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.361076 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5b9d0b_3470_4f33_bf0b_30c1cd569c42.slice/crio-c6f1c63ce8193a0c232c02f65ed5b3ec86407eb2307f71c3ecdf80abdffdafc4 WatchSource:0}: Error finding container c6f1c63ce8193a0c232c02f65ed5b3ec86407eb2307f71c3ecdf80abdffdafc4: Status 404 returned error can't find the container with id c6f1c63ce8193a0c232c02f65ed5b3ec86407eb2307f71c3ecdf80abdffdafc4 Jul 27 23:36:43.361997 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.361976 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3a0fc3_cab7_48ac_9e8d_e04de7418400.slice/crio-2ff447b42d62b21f5b3d6b6b9c128afb13f1539ecf9945a55a55e2cdfd3a8058 WatchSource:0}: Error finding container 2ff447b42d62b21f5b3d6b6b9c128afb13f1539ecf9945a55a55e2cdfd3a8058: Status 404 returned error can't find the container with id 2ff447b42d62b21f5b3d6b6b9c128afb13f1539ecf9945a55a55e2cdfd3a8058 Jul 27 23:36:43.362580 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.362550 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45760ac0_5980_4697_aaa1_c7945a19b317.slice/crio-cb6c937ea80a6c3b8a92d42054166c504ac8f92cafa01a16fc4d1867db94ded2 WatchSource:0}: Error finding container cb6c937ea80a6c3b8a92d42054166c504ac8f92cafa01a16fc4d1867db94ded2: Status 404 returned error can't find the container with id cb6c937ea80a6c3b8a92d42054166c504ac8f92cafa01a16fc4d1867db94ded2 Jul 27 23:36:43.363935 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.363913 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d212a78_26c3_46de_8dfe_f299488fdf2d.slice/crio-0dd99a8a9546ff57d9b27c8a2bc69580b5fafb00067802a1652df3f7c094a934 WatchSource:0}: Error finding container 0dd99a8a9546ff57d9b27c8a2bc69580b5fafb00067802a1652df3f7c094a934: Status 404 returned error can't find the container with id 0dd99a8a9546ff57d9b27c8a2bc69580b5fafb00067802a1652df3f7c094a934 Jul 27 23:36:43.365487 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.365082 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bb38c8_b12b_4248_af07_d67fc322c56a.slice/crio-f6475143245d6729be159b0d371159a33549831926142f19e63d171f04336ad1 WatchSource:0}: Error finding container f6475143245d6729be159b0d371159a33549831926142f19e63d171f04336ad1: Status 404 returned error can't find the container with id f6475143245d6729be159b0d371159a33549831926142f19e63d171f04336ad1 Jul 27 23:36:43.365871 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.365654 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3eb81d_5605_4d0a_a7ec_6e952d769f35.slice/crio-3b5aa3ff58d9ebc5ef35a82688d5c6404e386243673cc3d6e9478459812287ff WatchSource:0}: Error finding container 3b5aa3ff58d9ebc5ef35a82688d5c6404e386243673cc3d6e9478459812287ff: Status 404 returned error can't find the container with id 3b5aa3ff58d9ebc5ef35a82688d5c6404e386243673cc3d6e9478459812287ff Jul 27 23:36:43.367758 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.367304 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d14687_0f5b_47e0_9d9c_20c33d00e49e.slice/crio-295907f6ea53d486cdf6a612b130aa07d7f22752ac179b8444bffe88cf2d76ef WatchSource:0}: Error finding container 295907f6ea53d486cdf6a612b130aa07d7f22752ac179b8444bffe88cf2d76ef: Status 404 returned error can't find the container with id 295907f6ea53d486cdf6a612b130aa07d7f22752ac179b8444bffe88cf2d76ef Jul 27 23:36:43.368312 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:36:43.368290 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70962f98_5b88_4104_96d6_7fb957944574.slice/crio-c008eadb19c2413f92697b7c94bac4327c938eca1dc3a01b38882e6df65b24fa WatchSource:0}: Error finding container c008eadb19c2413f92697b7c94bac4327c938eca1dc3a01b38882e6df65b24fa: Status 404 returned error can't find the container with id c008eadb19c2413f92697b7c94bac4327c938eca1dc3a01b38882e6df65b24fa Jul 27 23:36:43.470175 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.470005 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:31:42 +0000 UTC" deadline="2028-01-11 04:32:11.507453429 +0000 UTC" Jul 27 23:36:43.470175 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.470175 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12772h55m28.037283114s" Jul 27 23:36:43.568258 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.568222 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" event={"ID":"a6d5157fea5c68626415a727705ed8a3","Type":"ContainerStarted","Data":"c60b139a400e0fbb6642d20af7ea4bcb7a08b8487b2d6f9ba62df45a92a0fa5d"} Jul 27 23:36:43.569543 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.569521 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"019df062654c9f634ff71e54cb9de65c","Type":"ContainerStarted","Data":"7cf5cc2afbf90163d06a035c6d744bf12b062375a669bd49bc694f3ffba5c2c1"} Jul 27 23:36:43.570448 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.570423 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n688k" event={"ID":"25d14687-0f5b-47e0-9d9c-20c33d00e49e","Type":"ContainerStarted","Data":"295907f6ea53d486cdf6a612b130aa07d7f22752ac179b8444bffe88cf2d76ef"} Jul 27 23:36:43.571423 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.571400 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s7jtw" event={"ID":"cc3eb81d-5605-4d0a-a7ec-6e952d769f35","Type":"ContainerStarted","Data":"3b5aa3ff58d9ebc5ef35a82688d5c6404e386243673cc3d6e9478459812287ff"} Jul 27 23:36:43.572477 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.572453 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rx964" event={"ID":"70962f98-5b88-4104-96d6-7fb957944574","Type":"ContainerStarted","Data":"c008eadb19c2413f92697b7c94bac4327c938eca1dc3a01b38882e6df65b24fa"} Jul 27 23:36:43.573512 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.573475 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"f6475143245d6729be159b0d371159a33549831926142f19e63d171f04336ad1"} Jul 27 23:36:43.574557 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.574537 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvzwz" event={"ID":"0d212a78-26c3-46de-8dfe-f299488fdf2d","Type":"ContainerStarted","Data":"0dd99a8a9546ff57d9b27c8a2bc69580b5fafb00067802a1652df3f7c094a934"} Jul 27 23:36:43.575484 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.575463 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k9p72" event={"ID":"45760ac0-5980-4697-aaa1-c7945a19b317","Type":"ContainerStarted","Data":"cb6c937ea80a6c3b8a92d42054166c504ac8f92cafa01a16fc4d1867db94ded2"} Jul 27 23:36:43.576453 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.576434 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerStarted","Data":"2ff447b42d62b21f5b3d6b6b9c128afb13f1539ecf9945a55a55e2cdfd3a8058"} Jul 27 23:36:43.577244 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.577225 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" event={"ID":"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42","Type":"ContainerStarted","Data":"c6f1c63ce8193a0c232c02f65ed5b3ec86407eb2307f71c3ecdf80abdffdafc4"} Jul 27 23:36:43.578571 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.578083 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cmgpx" event={"ID":"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e","Type":"ContainerStarted","Data":"29415ebdcf5b2d5e71a40b88941e6b7c292aafaf216d1805f311a8ef2fef463d"} Jul 27 23:36:43.582044 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:43.581957 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" podStartSLOduration=2.581943443 podStartE2EDuration="2.581943443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:36:43.580672375 +0000 UTC m=+3.504881276" watchObservedRunningTime="2026-07-27 23:36:43.581943443 +0000 UTC m=+3.506152320" Jul 27 23:36:44.057248 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.057161 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:44.057248 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.057223 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:44.057470 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.057368 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:44.057470 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.057431 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:46.057412146 +0000 UTC m=+5.981621049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:44.057605 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.057536 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:36:44.057605 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.057551 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:36:44.057605 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.057562 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:44.057605 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.057603 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:46.057591197 +0000 UTC m=+5.981800058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:44.158344 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.158306 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:44.158526 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.158514 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:44.158584 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.158572 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:46.158554094 +0000 UTC m=+6.082762955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:44.470812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.470567 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:31:42 +0000 UTC" deadline="2028-04-26 05:30:31.929023978 +0000 UTC" Jul 27 23:36:44.470812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.470622 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15317h53m47.458406197s" Jul 27 23:36:44.561688 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.561658 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:44.561856 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.561782 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:44.561964 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.561927 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:44.562053 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.562029 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:44.562118 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.562105 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:44.562208 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:44.562190 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:44.597658 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.597618 2555 generic.go:356] "Generic (PLEG): container finished" podID="019df062654c9f634ff71e54cb9de65c" containerID="7cf5cc2afbf90163d06a035c6d744bf12b062375a669bd49bc694f3ffba5c2c1" exitCode=0 Jul 27 23:36:44.597833 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:44.597720 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"019df062654c9f634ff71e54cb9de65c","Type":"ContainerDied","Data":"7cf5cc2afbf90163d06a035c6d744bf12b062375a669bd49bc694f3ffba5c2c1"} Jul 27 23:36:45.620575 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:45.620218 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"019df062654c9f634ff71e54cb9de65c","Type":"ContainerStarted","Data":"5af4b3e2e9b9eb1c19fd992c05cf23ff6b6dc0678a6dab1233c02164174f4b5c"} Jul 27 23:36:45.634057 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:45.634004 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" podStartSLOduration=4.633984641 podStartE2EDuration="4.633984641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:36:45.633663147 +0000 UTC m=+5.557872038" watchObservedRunningTime="2026-07-27 23:36:45.633984641 +0000 UTC m=+5.558193522" Jul 27 23:36:46.075915 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:46.075879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:46.076157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:46.075936 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:46.076157 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.076081 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:46.076157 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.076137 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:50.076119733 +0000 UTC m=+10.000328605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:46.076678 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.076562 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:36:46.076678 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.076585 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:36:46.076678 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.076598 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:46.076678 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.076644 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:50.076628945 +0000 UTC m=+10.000837808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:46.176907 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:46.176841 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:46.177083 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.176995 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:46.177083 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.177053 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:50.177036505 +0000 UTC m=+10.101245375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:46.559783 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:46.559705 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:46.559930 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.559862 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:46.560319 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:46.560292 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:46.560453 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.560409 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:46.560687 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:46.560545 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:46.560687 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:46.560646 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:48.559777 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:48.559739 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:48.560181 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:48.559874 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:48.560340 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:48.560323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:48.560455 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:48.560436 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:48.560598 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:48.560577 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:48.560706 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:48.560686 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:50.109472 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:50.109430 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:50.109926 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:50.109505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:50.109926 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.109645 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:50.109926 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.109707 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:58.109689405 +0000 UTC m=+18.033898276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:50.110142 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.110125 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:36:50.110190 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.110151 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:36:50.110190 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.110163 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:50.110249 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.110206 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:58.110191587 +0000 UTC m=+18.034400457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:50.211467 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:50.210892 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:50.211467 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.211059 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:50.211467 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.211123 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:36:58.211103751 +0000 UTC m=+18.135312611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:50.560229 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:50.560151 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:50.560395 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.560250 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:50.560395 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:50.560320 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:50.560533 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.560416 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:50.560533 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:50.560458 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:50.560645 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:50.560551 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:52.559705 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:52.559660 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:52.559705 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:52.559698 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:52.560140 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:52.559769 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:52.560140 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:52.559785 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:52.560140 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:52.559847 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:52.560140 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:52.559918 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:54.558936 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:54.558904 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:54.559384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:54.558908 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:54.559384 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:54.559022 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:54.559384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:54.559037 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:54.559384 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:54.559127 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:54.559384 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:54.559196 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:56.559529 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:56.559480 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:56.559987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:56.559480 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:56.559987 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:56.559638 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:56.559987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:56.559485 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:56.559987 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:56.559720 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:56.559987 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:56.559786 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:36:58.174227 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:58.174139 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:58.174227 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:58.174190 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:58.174745 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.174341 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:36:58.174745 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.174373 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:36:58.174745 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.174386 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:58.174745 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.174420 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:58.174745 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.174454 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:14.174435791 +0000 UTC m=+34.098644664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:36:58.174745 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.174486 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:14.174468856 +0000 UTC m=+34.098677736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:36:58.275130 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:58.275085 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:58.275316 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.275250 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:58.275372 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.275323 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:14.275303405 +0000 UTC m=+34.199512264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:36:58.559004 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:58.558972 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:36:58.559180 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:58.558971 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:36:58.559180 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.559107 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:36:58.559323 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.559189 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:36:58.559323 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:36:58.558981 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:36:58.559323 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:36:58.559284 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:00.560435 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:00.560401 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:00.560894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:00.560443 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:00.560894 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:00.560535 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:00.560894 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:00.560589 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:00.560894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:00.560645 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:00.560894 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:00.560711 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:02.559357 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.559002 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:02.560215 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.559002 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:02.560215 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:02.559456 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:02.560215 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:02.559605 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:02.560215 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.559057 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:02.560215 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:02.559698 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:02.656475 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.656441 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n688k" event={"ID":"25d14687-0f5b-47e0-9d9c-20c33d00e49e","Type":"ContainerStarted","Data":"7902180ad3724b2116152afac94f66bc276c356e6d4ff7ad813a5b5518194e9f"} Jul 27 23:37:02.658579 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.658527 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s7jtw" event={"ID":"cc3eb81d-5605-4d0a-a7ec-6e952d769f35","Type":"ContainerStarted","Data":"6746b4e7f9f3204ca3465facf105d3f10056cfc43c64feb270472d367c4f3efe"} Jul 27 23:37:02.660447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.660383 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rx964" event={"ID":"70962f98-5b88-4104-96d6-7fb957944574","Type":"ContainerStarted","Data":"d05caad84ab1c4e41634b3aa5174f733048bf1300383365db71a33b642ff88dd"} Jul 27 23:37:02.664059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.664038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"8dd5192ca33bcd1579f7937d73f5029dc7c8fa94964ac0a2609c55d7ac1e4352"} Jul 27 23:37:02.664159 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.664097 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"1ea4127d87f63cf8acd438795bb3a32ac5d95e2f5fdccf9da97fc7cd8bc433ab"} Jul 27 23:37:02.664159 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.664138 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"03160021d6cc8104e96596c4b24685b9e12a60a5b47efdf1ea3017cc5afd0f1d"} Jul 27 23:37:02.664259 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.664186 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"11e71623cec7f54a1454be9cd3f576d863d1faed047ffedc54973e6f5b7c5568"} Jul 27 23:37:02.664259 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.664232 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"309549ca90a261ee3c1395f04c2ac96993a54065429d6527be24df96eb196587"} Jul 27 23:37:02.665628 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.665605 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k9p72" event={"ID":"45760ac0-5980-4697-aaa1-c7945a19b317","Type":"ContainerStarted","Data":"91991f67fe4652c552b472f291d206630d3799e22eb6f52a03c226215e3e6acb"} Jul 27 23:37:02.667364 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.667292 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a3a0fc3-cab7-48ac-9e8d-e04de7418400" containerID="6aa8facaab9050b03b48f6c7025a4e6167d5b1049b81f6334f7e060f892bab12" exitCode=0 Jul 27 23:37:02.667469 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.667378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerDied","Data":"6aa8facaab9050b03b48f6c7025a4e6167d5b1049b81f6334f7e060f892bab12"} Jul 27 23:37:02.669056 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.669006 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" event={"ID":"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42","Type":"ContainerStarted","Data":"e9d12f1e255fb6586fe0eed7477358e4fb93a8b28439e50b6c46a7a79eb28de5"} Jul 27 23:37:02.670800 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.670718 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cmgpx" event={"ID":"2d50a704-4e89-4af2-ad0f-d7ad884eaa1e","Type":"ContainerStarted","Data":"8618db14ce2466b1e66e23a49f100b2264696cb7e7a768aaaa25819a9630c870"} Jul 27 23:37:02.671880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.671846 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n688k" podStartSLOduration=4.10889561 podStartE2EDuration="22.671834635s" totalImagesPullingTime="18.562939025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:02.671304125 +0000 UTC m=+22.595513005" watchObservedRunningTime="2026-07-27 23:37:02.671834635 +0000 UTC m=+22.596043512" Jul 27 23:37:02.705388 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.705341 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k9p72" podStartSLOduration=4.079236257 podStartE2EDuration="22.705325588s" totalImagesPullingTime="18.626089331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:02.705280521 +0000 UTC m=+22.629489401" watchObservedRunningTime="2026-07-27 23:37:02.705325588 +0000 UTC m=+22.629534463" Jul 27 23:37:02.720237 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.720192 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rx964" podStartSLOduration=4.143366399 podStartE2EDuration="22.720178617s" totalImagesPullingTime="18.576812218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:02.720015604 +0000 UTC m=+22.644224481" watchObservedRunningTime="2026-07-27 23:37:02.720178617 +0000 UTC m=+22.644387492" Jul 27 23:37:02.738723 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.738670 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s7jtw" podStartSLOduration=4.174538189 podStartE2EDuration="22.738656869s" totalImagesPullingTime="18.56411868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:02.738637832 +0000 UTC m=+22.662846710" watchObservedRunningTime="2026-07-27 23:37:02.738656869 +0000 UTC m=+22.662865748" Jul 27 23:37:02.751982 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:02.751935 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cmgpx" podStartSLOduration=8.413062568 podStartE2EDuration="22.751922016s" totalImagesPullingTime="14.338859448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:02.751508217 +0000 UTC m=+22.675717089" watchObservedRunningTime="2026-07-27 23:37:02.751922016 +0000 UTC m=+22.676130894" Jul 27 23:37:03.401956 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.401926 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 23:37:03.488745 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.488553 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T23:37:03.40195245Z","UUID":"5840846b-f83a-4082-88ec-5b72f68a8739","Handler":null,"Name":"","Endpoint":""} Jul 27 23:37:03.490227 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.490199 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 23:37:03.490334 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.490238 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 23:37:03.676807 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.676771 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"c66ee17932a3dfaed37877f6d2e6e8ad4034cd4c3fe697e7fbc4c6c7099ca5aa"} Jul 27 23:37:03.678786 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.678677 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvzwz" event={"ID":"0d212a78-26c3-46de-8dfe-f299488fdf2d","Type":"ContainerStarted","Data":"3baab902362ccad3e3471b4baa981b922bbc19e4f8095ff7737afe8d55acb0f7"} Jul 27 23:37:03.680920 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.680896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" event={"ID":"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42","Type":"ContainerStarted","Data":"fafba720b7de39e01dfb84f077d00ffe3568c36511a4809b7db31d389002ccac"} Jul 27 23:37:03.691834 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:03.691780 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jvzwz" podStartSLOduration=5.124833674 podStartE2EDuration="23.69176703s" totalImagesPullingTime="18.566933356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:03.691527593 +0000 UTC m=+23.615736472" watchObservedRunningTime="2026-07-27 23:37:03.69176703 +0000 UTC m=+23.615975907" Jul 27 23:37:04.559517 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:04.559392 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:04.559517 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:04.559436 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:04.559753 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:04.559584 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:04.559913 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:04.559895 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:04.560070 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:04.559995 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:04.560070 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:04.560057 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:04.685200 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:04.685151 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" event={"ID":"bf5b9d0b-3470-4f33-bf0b-30c1cd569c42","Type":"ContainerStarted","Data":"13a0c8c13a051eaf8f8223c9c62a22dc72f14f99e928f139f85255395cc6c7e8"} Jul 27 23:37:04.700088 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:04.700039 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ff5gk" podStartSLOduration=3.877052413 podStartE2EDuration="24.700020955s" totalImagesPullingTime="20.822968542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:04.699429918 +0000 UTC m=+24.623638796" watchObservedRunningTime="2026-07-27 23:37:04.700020955 +0000 UTC m=+24.624229832" Jul 27 23:37:05.694301 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:05.694050 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"f37b74475e08e96eb7d700b482613b90faea1f88d8bb64f39870bb726f703e51"} Jul 27 23:37:06.282512 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.282468 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:37:06.283074 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.283051 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:37:06.559416 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.559326 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:06.559416 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.559371 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:06.559638 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.559326 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:06.559638 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:06.559467 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:06.559638 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:06.559549 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:06.559638 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:06.559611 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:06.696946 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.696910 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:37:06.697454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:06.697381 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cmgpx" Jul 27 23:37:07.702509 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:07.702459 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" event={"ID":"18bb38c8-b12b-4248-af07-d67fc322c56a","Type":"ContainerStarted","Data":"9fde14ca22c0342e41b900d8feca036ff9705faeed3aa7760a8e850bf78179e5"} Jul 27 23:37:07.703238 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:07.702793 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:37:07.704404 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:07.704369 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a3a0fc3-cab7-48ac-9e8d-e04de7418400" containerID="e40d56deebe5afd102d6709a6434416a962468ed1557e6be2660a6b4b42c452d" exitCode=0 Jul 27 23:37:07.704528 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:07.704449 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerDied","Data":"e40d56deebe5afd102d6709a6434416a962468ed1557e6be2660a6b4b42c452d"} Jul 27 23:37:07.718844 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:07.718824 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:37:07.729311 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:07.729271 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" podStartSLOduration=9.099565952 podStartE2EDuration="27.729258438s" totalImagesPullingTime="18.629692486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:07.728854041 +0000 UTC m=+27.653062918" watchObservedRunningTime="2026-07-27 23:37:07.729258438 +0000 UTC m=+27.653467315" Jul 27 23:37:08.559741 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:08.559712 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:08.559997 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:08.559713 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:08.559997 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:08.559832 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:08.559997 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:08.559713 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:08.559997 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:08.559916 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:08.559997 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:08.559967 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:08.706916 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:08.706738 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:37:08.706916 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:08.706822 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:37:08.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:08.723810 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:37:09.253382 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.253348 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqttn"] Jul 27 23:37:09.253560 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.253458 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:09.253610 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:09.253582 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:09.256753 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.256723 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ppxkd"] Jul 27 23:37:09.256915 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.256831 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:09.256978 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:09.256915 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:09.257340 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.257317 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89dpp"] Jul 27 23:37:09.257432 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.257405 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:09.257539 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:09.257518 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:09.711280 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.711249 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a3a0fc3-cab7-48ac-9e8d-e04de7418400" containerID="57c2c9f84797895198de58e01cfdd785ac3712a8d682d51e55e660345e488b23" exitCode=0 Jul 27 23:37:09.711679 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:09.711332 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerDied","Data":"57c2c9f84797895198de58e01cfdd785ac3712a8d682d51e55e660345e488b23"} Jul 27 23:37:10.562432 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:10.562257 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:10.562623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:10.562323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:10.562623 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:10.562531 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:10.562623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:10.562324 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:10.562623 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:10.562609 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:10.562824 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:10.562653 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:11.718605 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:11.718572 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a3a0fc3-cab7-48ac-9e8d-e04de7418400" containerID="adb0e5776104d7abb58702cc9b3e9d549e5d2761b1e7b82189cdee51c2f4d1bf" exitCode=0 Jul 27 23:37:11.719112 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:11.718612 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerDied","Data":"adb0e5776104d7abb58702cc9b3e9d549e5d2761b1e7b82189cdee51c2f4d1bf"} Jul 27 23:37:12.559604 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:12.559566 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:12.559795 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:12.559566 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:12.559795 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:12.559691 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:12.559795 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:12.559779 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:12.559958 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:12.559571 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:12.559958 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:12.559894 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:14.187815 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:14.187777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:14.187826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.187937 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.187951 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.187960 2555 projected.go:196] Error preparing data for projected volume kube-api-access-p7dpt for pod openshift-network-diagnostics/network-check-target-89dpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.187987 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.188019 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt podName:faafc2d2-a65b-4cfd-990b-7da73a6454f4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:46.188000765 +0000 UTC m=+66.112209640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p7dpt" (UniqueName: "kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt") pod "network-check-target-89dpp" (UID: "faafc2d2-a65b-4cfd-990b-7da73a6454f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:37:14.188292 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.188048 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret podName:5ab68dc8-3b95-4c0e-9a6c-611abfc63751 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:46.188032293 +0000 UTC m=+66.112241166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret") pod "global-pull-secret-syncer-gqttn" (UID: "5ab68dc8-3b95-4c0e-9a6c-611abfc63751") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:37:14.288486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:14.288447 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:14.288678 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.288587 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:37:14.288678 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.288652 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs podName:3e25d840-3aaa-417d-9ede-4a6e9f79d730 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:46.288636442 +0000 UTC m=+66.212845300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs") pod "network-metrics-daemon-ppxkd" (UID: "3e25d840-3aaa-417d-9ede-4a6e9f79d730") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:37:14.563161 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:14.563128 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:14.563347 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:14.563198 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:14.563347 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.563306 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ppxkd" podUID="3e25d840-3aaa-417d-9ede-4a6e9f79d730" Jul 27 23:37:14.563347 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:14.563319 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:14.563521 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.563399 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89dpp" podUID="faafc2d2-a65b-4cfd-990b-7da73a6454f4" Jul 27 23:37:14.563521 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:14.563468 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqttn" podUID="5ab68dc8-3b95-4c0e-9a6c-611abfc63751" Jul 27 23:37:15.864335 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.864313 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeReady" Jul 27 23:37:15.864800 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.864447 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 23:37:15.897109 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.897081 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-796945c6c4-jrxlv"] Jul 27 23:37:15.898941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.898925 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:15.900982 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.900962 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 23:37:15.901124 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.901105 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 23:37:15.901218 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.901203 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 23:37:15.901352 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.901333 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mkzjz\"" Jul 27 23:37:15.904640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.904624 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 23:37:15.909243 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.909226 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lqr4s"] Jul 27 23:37:15.910799 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.910786 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b2fvf"] Jul 27 23:37:15.910915 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.910900 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:15.912322 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.912302 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:15.912574 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.912551 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-796945c6c4-jrxlv"] Jul 27 23:37:15.912987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.912967 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 23:37:15.913105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.913032 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jk8w\"" Jul 27 23:37:15.913105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.912973 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 23:37:15.913990 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.913974 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fz2fq\"" Jul 27 23:37:15.914299 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.914276 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 23:37:15.914299 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.914292 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 23:37:15.914425 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.914292 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 23:37:15.923263 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.923186 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2fvf"] Jul 27 23:37:15.924030 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:15.924004 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqr4s"] Jul 27 23:37:16.000988 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.000951 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a89c6a8f-50c8-46bb-ab07-ac6663f43fd9-cert\") pod \"ingress-canary-b2fvf\" (UID: \"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9\") " pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.000994 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-image-registry-private-configuration\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001021 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-registry-certificates\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001038 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-installation-pull-secrets\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001081 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001112 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65b59dc8-4046-499e-97e0-3192fa260310-config-volume\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001132 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9288t\" (UniqueName: \"kubernetes.io/projected/a89c6a8f-50c8-46bb-ab07-ac6663f43fd9-kube-api-access-9288t\") pod \"ingress-canary-b2fvf\" (UID: \"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9\") " pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.001157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001154 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001169 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpg4j\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-kube-api-access-wpg4j\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001184 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxq6\" (UniqueName: \"kubernetes.io/projected/65b59dc8-4046-499e-97e0-3192fa260310-kube-api-access-jqxq6\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.001461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001230 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-trusted-ca\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001275 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b59dc8-4046-499e-97e0-3192fa260310-tmp-dir\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.001461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47955614-5597-4835-9f17-48edb322ded4-ca-trust-extracted\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.001461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.001384 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-bound-sa-token\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102260 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102224 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a89c6a8f-50c8-46bb-ab07-ac6663f43fd9-cert\") pod \"ingress-canary-b2fvf\" (UID: \"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9\") " pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.102410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102269 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-image-registry-private-configuration\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102300 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-registry-certificates\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102353 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-installation-pull-secrets\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65b59dc8-4046-499e-97e0-3192fa260310-config-volume\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102438 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9288t\" (UniqueName: \"kubernetes.io/projected/a89c6a8f-50c8-46bb-ab07-ac6663f43fd9-kube-api-access-9288t\") pod \"ingress-canary-b2fvf\" (UID: \"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9\") " pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102465 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102486 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpg4j\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-kube-api-access-wpg4j\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102532 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxq6\" (UniqueName: \"kubernetes.io/projected/65b59dc8-4046-499e-97e0-3192fa260310-kube-api-access-jqxq6\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.102564 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.102601 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:37:16.102615 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.102620 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-796945c6c4-jrxlv: secret "image-registry-tls" not found Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.102637 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls podName:65b59dc8-4046-499e-97e0-3192fa260310 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:16.602615691 +0000 UTC m=+36.526824559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls") pod "dns-default-lqr4s" (UID: "65b59dc8-4046-499e-97e0-3192fa260310") : secret "dns-default-metrics-tls" not found Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.102667 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls podName:47955614-5597-4835-9f17-48edb322ded4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:16.602651425 +0000 UTC m=+36.526860298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls") pod "image-registry-796945c6c4-jrxlv" (UID: "47955614-5597-4835-9f17-48edb322ded4") : secret "image-registry-tls" not found Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102720 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-trusted-ca\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102759 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b59dc8-4046-499e-97e0-3192fa260310-tmp-dir\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102829 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47955614-5597-4835-9f17-48edb322ded4-ca-trust-extracted\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102872 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-bound-sa-token\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.102977 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.102920 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-registry-certificates\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.103302 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.103082 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65b59dc8-4046-499e-97e0-3192fa260310-config-volume\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.103302 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.103150 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b59dc8-4046-499e-97e0-3192fa260310-tmp-dir\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.103572 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.103551 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47955614-5597-4835-9f17-48edb322ded4-ca-trust-extracted\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.103631 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.103615 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-trusted-ca\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.106777 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.106653 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a89c6a8f-50c8-46bb-ab07-ac6663f43fd9-cert\") pod \"ingress-canary-b2fvf\" (UID: \"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9\") " pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.106878 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.106661 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-image-registry-private-configuration\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.106878 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.106687 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-installation-pull-secrets\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.111255 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.111234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-bound-sa-token\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.111384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.111266 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxq6\" (UniqueName: \"kubernetes.io/projected/65b59dc8-4046-499e-97e0-3192fa260310-kube-api-access-jqxq6\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.111513 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.111479 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpg4j\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-kube-api-access-wpg4j\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.111855 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.111834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9288t\" (UniqueName: \"kubernetes.io/projected/a89c6a8f-50c8-46bb-ab07-ac6663f43fd9-kube-api-access-9288t\") pod \"ingress-canary-b2fvf\" (UID: \"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9\") " pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.226217 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.226165 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2fvf" Jul 27 23:37:16.354693 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.354663 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2fvf"] Jul 27 23:37:16.358508 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:16.358468 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89c6a8f_50c8_46bb_ab07_ac6663f43fd9.slice/crio-5e8be6bf055de23dc02cab8613ae28d2a0541ee649cc3dc6bdb27709b6b04736 WatchSource:0}: Error finding container 5e8be6bf055de23dc02cab8613ae28d2a0541ee649cc3dc6bdb27709b6b04736: Status 404 returned error can't find the container with id 5e8be6bf055de23dc02cab8613ae28d2a0541ee649cc3dc6bdb27709b6b04736 Jul 27 23:37:16.562802 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.562679 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:16.562952 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.562820 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:16.563013 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.562820 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:16.565172 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.565144 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 23:37:16.565172 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.565169 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 23:37:16.565354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.565200 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8p7f9\"" Jul 27 23:37:16.565354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.565200 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 23:37:16.565474 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.565460 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 23:37:16.565554 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.565544 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t92bh\"" Jul 27 23:37:16.606940 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.606910 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:16.607083 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.606952 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:16.607083 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.607042 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:37:16.607083 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.607055 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-796945c6c4-jrxlv: secret "image-registry-tls" not found Jul 27 23:37:16.607173 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.607106 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls podName:47955614-5597-4835-9f17-48edb322ded4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:17.607091984 +0000 UTC m=+37.531300841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls") pod "image-registry-796945c6c4-jrxlv" (UID: "47955614-5597-4835-9f17-48edb322ded4") : secret "image-registry-tls" not found Jul 27 23:37:16.607173 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.607115 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:37:16.607173 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:16.607167 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls podName:65b59dc8-4046-499e-97e0-3192fa260310 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:17.607150784 +0000 UTC m=+37.531359643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls") pod "dns-default-lqr4s" (UID: "65b59dc8-4046-499e-97e0-3192fa260310") : secret "dns-default-metrics-tls" not found Jul 27 23:37:16.731673 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.731637 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2fvf" event={"ID":"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9","Type":"ContainerStarted","Data":"5e8be6bf055de23dc02cab8613ae28d2a0541ee649cc3dc6bdb27709b6b04736"} Jul 27 23:37:16.734976 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.734945 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a3a0fc3-cab7-48ac-9e8d-e04de7418400" containerID="81ee69597543973cf61516232c02ffbc520af2b967a18d7858788d163cc3c328" exitCode=0 Jul 27 23:37:16.735102 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:16.734979 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerDied","Data":"81ee69597543973cf61516232c02ffbc520af2b967a18d7858788d163cc3c328"} Jul 27 23:37:17.616719 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:17.616679 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:17.617191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:17.616736 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:17.617191 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:17.616893 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:37:17.617191 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:17.616959 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:37:17.617191 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:17.616999 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-796945c6c4-jrxlv: secret "image-registry-tls" not found Jul 27 23:37:17.617191 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:17.616979 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls podName:65b59dc8-4046-499e-97e0-3192fa260310 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:19.616956049 +0000 UTC m=+39.541164918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls") pod "dns-default-lqr4s" (UID: "65b59dc8-4046-499e-97e0-3192fa260310") : secret "dns-default-metrics-tls" not found Jul 27 23:37:17.617191 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:17.617053 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls podName:47955614-5597-4835-9f17-48edb322ded4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:19.617041251 +0000 UTC m=+39.541250108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls") pod "image-registry-796945c6c4-jrxlv" (UID: "47955614-5597-4835-9f17-48edb322ded4") : secret "image-registry-tls" not found Jul 27 23:37:17.741466 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:17.741204 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a3a0fc3-cab7-48ac-9e8d-e04de7418400" containerID="fd8ba2bc46baa8a1a211549a61a7ced1e30cf47d5c3587acb4810d0c77e49b03" exitCode=0 Jul 27 23:37:17.741632 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:17.741278 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerDied","Data":"fd8ba2bc46baa8a1a211549a61a7ced1e30cf47d5c3587acb4810d0c77e49b03"} Jul 27 23:37:18.746046 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:18.746018 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" event={"ID":"3a3a0fc3-cab7-48ac-9e8d-e04de7418400","Type":"ContainerStarted","Data":"5086f840d924661822585efd1bc1b8c4b3c517e8139f97b722ad68e95509df4e"} Jul 27 23:37:18.770147 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:18.770102 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pm6m5" podStartSLOduration=9.954346815 podStartE2EDuration="38.770086918s" totalImagesPullingTime="28.815740103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:18.769242597 +0000 UTC m=+38.693451474" watchObservedRunningTime="2026-07-27 23:37:18.770086918 +0000 UTC m=+38.694295792" Jul 27 23:37:18.915039 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:18.914998 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:18.993434 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:18.993400 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:19.243230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:19.243192 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:19.536816 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:19.536781 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:19.634587 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:19.634552 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:19.634764 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:19.634605 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:19.634764 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:19.634712 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:37:19.634923 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:19.634715 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:37:19.634923 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:19.634791 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-796945c6c4-jrxlv: secret "image-registry-tls" not found Jul 27 23:37:19.634923 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:19.634776 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls podName:65b59dc8-4046-499e-97e0-3192fa260310 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:23.634760916 +0000 UTC m=+43.558969771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls") pod "dns-default-lqr4s" (UID: "65b59dc8-4046-499e-97e0-3192fa260310") : secret "dns-default-metrics-tls" not found Jul 27 23:37:19.634923 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:19.634845 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls podName:47955614-5597-4835-9f17-48edb322ded4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:23.634833079 +0000 UTC m=+43.559041935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls") pod "image-registry-796945c6c4-jrxlv" (UID: "47955614-5597-4835-9f17-48edb322ded4") : secret "image-registry-tls" not found Jul 27 23:37:19.753071 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:19.753036 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2fvf" event={"ID":"a89c6a8f-50c8-46bb-ab07-ac6663f43fd9","Type":"ContainerStarted","Data":"e5b90453b9e00d63f11e557b5035b99411231dc506d28b93c9fd8e5b62963f0a"} Jul 27 23:37:19.768691 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:19.768643 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b2fvf" podStartSLOduration=2.400390834 podStartE2EDuration="4.76862802s" totalImagesPullingTime="2.368237186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:19.767932846 +0000 UTC m=+39.692141722" watchObservedRunningTime="2026-07-27 23:37:19.76862802 +0000 UTC m=+39.692836897" Jul 27 23:37:23.664174 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:23.664135 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:23.664593 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:23.664181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:23.664593 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:23.664285 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:37:23.664593 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:23.664299 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:37:23.664593 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:23.664309 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-796945c6c4-jrxlv: secret "image-registry-tls" not found Jul 27 23:37:23.664593 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:23.664352 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls podName:65b59dc8-4046-499e-97e0-3192fa260310 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:31.66433523 +0000 UTC m=+51.588544086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls") pod "dns-default-lqr4s" (UID: "65b59dc8-4046-499e-97e0-3192fa260310") : secret "dns-default-metrics-tls" not found Jul 27 23:37:23.664593 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:23.664366 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls podName:47955614-5597-4835-9f17-48edb322ded4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:31.66435977 +0000 UTC m=+51.588568625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls") pod "image-registry-796945c6c4-jrxlv" (UID: "47955614-5597-4835-9f17-48edb322ded4") : secret "image-registry-tls" not found Jul 27 23:37:24.561874 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:24.561837 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:24.660421 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:24.660375 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:24.760291 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:24.760254 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:24.917589 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:24.917488 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:25.370239 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.370205 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:25.393511 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.393477 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg"] Jul 27 23:37:25.428754 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.428720 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg"] Jul 27 23:37:25.428900 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.428845 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" Jul 27 23:37:25.431408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.431383 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 23:37:25.431538 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.431478 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lbnc5\"" Jul 27 23:37:25.432098 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.432059 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 23:37:25.446216 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.446188 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:25.506139 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.506084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:25.578723 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.578694 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvwg\" (UniqueName: \"kubernetes.io/projected/80811de6-0dbd-4dad-a545-7b4efd4097c0-kube-api-access-drvwg\") pod \"migrator-6865c9967-c79pg\" (UID: \"80811de6-0dbd-4dad-a545-7b4efd4097c0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" Jul 27 23:37:25.618022 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.617987 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:25.670990 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.670903 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:25.679891 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.679861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvwg\" (UniqueName: \"kubernetes.io/projected/80811de6-0dbd-4dad-a545-7b4efd4097c0-kube-api-access-drvwg\") pod \"migrator-6865c9967-c79pg\" (UID: \"80811de6-0dbd-4dad-a545-7b4efd4097c0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" Jul 27 23:37:25.688941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.688918 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvwg\" (UniqueName: \"kubernetes.io/projected/80811de6-0dbd-4dad-a545-7b4efd4097c0-kube-api-access-drvwg\") pod \"migrator-6865c9967-c79pg\" (UID: \"80811de6-0dbd-4dad-a545-7b4efd4097c0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" Jul 27 23:37:25.737391 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.737366 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" Jul 27 23:37:25.874975 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:25.874943 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg"] Jul 27 23:37:25.878161 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:25.878134 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80811de6_0dbd_4dad_a545_7b4efd4097c0.slice/crio-c1e23c1697d9c0d2c7db23612456b6fa7c9ca0498551a8ea3dd2277e011bb9fd WatchSource:0}: Error finding container c1e23c1697d9c0d2c7db23612456b6fa7c9ca0498551a8ea3dd2277e011bb9fd: Status 404 returned error can't find the container with id c1e23c1697d9c0d2c7db23612456b6fa7c9ca0498551a8ea3dd2277e011bb9fd Jul 27 23:37:26.230809 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.230769 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-46fmv"] Jul 27 23:37:26.246883 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.246838 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:26.273972 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.273940 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-46fmv"] Jul 27 23:37:26.274118 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.274072 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.276015 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.275990 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 23:37:26.276015 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.275997 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 23:37:26.276212 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.276071 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 23:37:26.276803 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.276786 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 23:37:26.276803 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.276802 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sbzb5\"" Jul 27 23:37:26.309776 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.309740 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:26.385362 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.385328 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/524dce36-43f9-45c7-bf24-6ad7918aeb94-crio-socket\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.385362 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.385361 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.385567 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.385404 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/524dce36-43f9-45c7-bf24-6ad7918aeb94-data-volume\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.385567 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.385468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.385567 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.385483 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5hx\" (UniqueName: \"kubernetes.io/projected/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-api-access-dl5hx\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486046 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.485963 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486046 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.485995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5hx\" (UniqueName: \"kubernetes.io/projected/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-api-access-dl5hx\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486046 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.486027 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/524dce36-43f9-45c7-bf24-6ad7918aeb94-crio-socket\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.486054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.486086 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.486110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/524dce36-43f9-45c7-bf24-6ad7918aeb94-data-volume\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.486167 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:26.986151645 +0000 UTC m=+46.910360520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : configmap "kube-rbac-proxy" not found Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.486220 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.486248 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/524dce36-43f9-45c7-bf24-6ad7918aeb94-crio-socket\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.486299 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.486281 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:26.98626525 +0000 UTC m=+46.910474115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : secret "insights-runtime-extractor-tls" not found Jul 27 23:37:26.486556 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.486447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/524dce36-43f9-45c7-bf24-6ad7918aeb94-data-volume\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.493778 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.493761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5hx\" (UniqueName: \"kubernetes.io/projected/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-api-access-dl5hx\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.769817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.769778 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" event={"ID":"80811de6-0dbd-4dad-a545-7b4efd4097c0","Type":"ContainerStarted","Data":"c1e23c1697d9c0d2c7db23612456b6fa7c9ca0498551a8ea3dd2277e011bb9fd"} Jul 27 23:37:26.990267 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.990228 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.990640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:26.990275 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:26.990640 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.990349 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:37:26.990640 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.990377 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:37:26.990640 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.990415 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:27.990398269 +0000 UTC m=+47.914607125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : configmap "kube-rbac-proxy" not found Jul 27 23:37:26.990640 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:26.990430 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:27.990424523 +0000 UTC m=+47.914633379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : secret "insights-runtime-extractor-tls" not found Jul 27 23:37:27.214623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.214541 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mxht6"] Jul 27 23:37:27.220932 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.220909 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.223202 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.223178 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 23:37:27.223355 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.223303 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nt52l\"" Jul 27 23:37:27.224405 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.223865 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 23:37:27.224405 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.223986 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 23:37:27.224405 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.224130 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 23:37:27.225170 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.225131 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mxht6"] Jul 27 23:37:27.297035 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.297002 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.348528 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.348480 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.393164 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.393122 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9hn5\" (UniqueName: \"kubernetes.io/projected/73978386-4e06-41f7-8927-f492d1b5f201-kube-api-access-s9hn5\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.393353 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.393212 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73978386-4e06-41f7-8927-f492d1b5f201-signing-key\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.393353 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.393237 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73978386-4e06-41f7-8927-f492d1b5f201-signing-cabundle\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.419380 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.419342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.494540 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.494511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hn5\" (UniqueName: \"kubernetes.io/projected/73978386-4e06-41f7-8927-f492d1b5f201-kube-api-access-s9hn5\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.494692 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.494562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73978386-4e06-41f7-8927-f492d1b5f201-signing-key\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.494692 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.494579 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73978386-4e06-41f7-8927-f492d1b5f201-signing-cabundle\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.495259 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.495241 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73978386-4e06-41f7-8927-f492d1b5f201-signing-cabundle\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.496941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.496922 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73978386-4e06-41f7-8927-f492d1b5f201-signing-key\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.501792 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.501770 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hn5\" (UniqueName: \"kubernetes.io/projected/73978386-4e06-41f7-8927-f492d1b5f201-kube-api-access-s9hn5\") pod \"service-ca-658d7b7c78-mxht6\" (UID: \"73978386-4e06-41f7-8927-f492d1b5f201\") " pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.521365 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.521327 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.532190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.532167 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" Jul 27 23:37:27.721024 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.720973 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.792914 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.792883 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mxht6"] Jul 27 23:37:27.796053 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:27.796007 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73978386_4e06_41f7_8927_f492d1b5f201.slice/crio-fcd2fd3c38f844f8b0590d5e484c0aac41c8dd3a226d0fc870b6d54a90783a1c WatchSource:0}: Error finding container fcd2fd3c38f844f8b0590d5e484c0aac41c8dd3a226d0fc870b6d54a90783a1c: Status 404 returned error can't find the container with id fcd2fd3c38f844f8b0590d5e484c0aac41c8dd3a226d0fc870b6d54a90783a1c Jul 27 23:37:27.796423 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.796391 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.870365 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.870320 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:27.999559 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.999520 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:27.999951 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:27.999574 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:27.999951 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:27.999658 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:37:27.999951 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:27.999690 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:37:27.999951 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:27.999738 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:29.999716277 +0000 UTC m=+49.923925134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : configmap "kube-rbac-proxy" not found Jul 27 23:37:27.999951 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:27.999759 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:29.999749647 +0000 UTC m=+49.923958507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : secret "insights-runtime-extractor-tls" not found Jul 27 23:37:28.124740 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:28.124655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:28.395846 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:28.395768 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n688k_25d14687-0f5b-47e0-9d9c-20c33d00e49e/dns-node-resolver/0.log" Jul 27 23:37:28.776694 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:28.776648 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" event={"ID":"80811de6-0dbd-4dad-a545-7b4efd4097c0","Type":"ContainerStarted","Data":"bda9158d5093c545eb746e44f8ee764ee6f4b308856ece35e48bfb5638fef32c"} Jul 27 23:37:28.776939 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:28.776716 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" event={"ID":"80811de6-0dbd-4dad-a545-7b4efd4097c0","Type":"ContainerStarted","Data":"553170f3b791692f9ad4a97e3db59158eea2f263074f080678bd3797cd1d7e30"} Jul 27 23:37:28.777850 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:28.777824 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" event={"ID":"73978386-4e06-41f7-8927-f492d1b5f201","Type":"ContainerStarted","Data":"fcd2fd3c38f844f8b0590d5e484c0aac41c8dd3a226d0fc870b6d54a90783a1c"} Jul 27 23:37:28.790410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:28.790358 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c79pg" podStartSLOduration=2.009031511 podStartE2EDuration="3.790341436s" totalImagesPullingTime="1.781309925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:28.790058293 +0000 UTC m=+48.714267173" watchObservedRunningTime="2026-07-27 23:37:28.790341436 +0000 UTC m=+48.714550331" Jul 27 23:37:29.794525 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:29.794476 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s7jtw_cc3eb81d-5605-4d0a-a7ec-6e952d769f35/node-ca/0.log" Jul 27 23:37:30.015818 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.015789 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:30.015931 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.015831 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:30.015931 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:30.015910 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:37:30.015931 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:30.015927 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:37:30.016067 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:30.015979 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:34.015964082 +0000 UTC m=+53.940172937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : secret "insights-runtime-extractor-tls" not found Jul 27 23:37:30.016067 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:30.015993 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:34.015987136 +0000 UTC m=+53.940195992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : configmap "kube-rbac-proxy" not found Jul 27 23:37:30.394987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.394956 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2fvf_a89c6a8f-50c8-46bb-ab07-ac6663f43fd9/serve-healthcheck-canary/0.log" Jul 27 23:37:30.484159 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.484100 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:30.595660 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.595627 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c79pg_80811de6-0dbd-4dad-a545-7b4efd4097c0/migrator/0.log" Jul 27 23:37:30.784450 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.784414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" event={"ID":"73978386-4e06-41f7-8927-f492d1b5f201","Type":"ContainerStarted","Data":"038f5de4bf2952dc549ba1f6092ac0dba2c37f00d5ad3aa8eb233f2467bed987"} Jul 27 23:37:30.794231 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.794208 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c79pg_80811de6-0dbd-4dad-a545-7b4efd4097c0/graceful-termination/0.log" Jul 27 23:37:30.798461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.798422 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-mxht6" podStartSLOduration=1.658318081 podStartE2EDuration="3.798407262s" totalImagesPullingTime="2.140089181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:30.797593941 +0000 UTC m=+50.721802820" watchObservedRunningTime="2026-07-27 23:37:30.798407262 +0000 UTC m=+50.722616140" Jul 27 23:37:30.891470 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:30.891425 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:31.484611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:31.484575 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:31.740656 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:31.740622 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:31.740840 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:31.740693 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:31.740901 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:31.740870 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:37:31.740901 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:31.740884 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-796945c6c4-jrxlv: secret "image-registry-tls" not found Jul 27 23:37:31.741000 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:31.740943 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls podName:47955614-5597-4835-9f17-48edb322ded4 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:47.740924923 +0000 UTC m=+67.665133786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls") pod "image-registry-796945c6c4-jrxlv" (UID: "47955614-5597-4835-9f17-48edb322ded4") : secret "image-registry-tls" not found Jul 27 23:37:31.741208 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:31.741189 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:37:31.741348 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:31.741336 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls podName:65b59dc8-4046-499e-97e0-3192fa260310 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:47.74131902 +0000 UTC m=+67.665527891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls") pod "dns-default-lqr4s" (UID: "65b59dc8-4046-499e-97e0-3192fa260310") : secret "dns-default-metrics-tls" not found Jul 27 23:37:31.836408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:31.836359 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:31.936615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:31.936568 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:32.037818 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:32.037734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:32.396483 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:32.396386 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:32.794260 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:32.794219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:33.394306 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:33.394272 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:33.788802 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:33.788767 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:34.058529 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:34.058415 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:34.058529 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:34.058469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:34.058715 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:34.058660 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:37:34.058750 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:34.058729 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls podName:524dce36-43f9-45c7-bf24-6ad7918aeb94 nodeName:}" failed. No retries permitted until 2026-07-27 23:37:42.05871301 +0000 UTC m=+61.982921866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46fmv" (UID: "524dce36-43f9-45c7-bf24-6ad7918aeb94") : secret "insights-runtime-extractor-tls" not found Jul 27 23:37:34.058998 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:34.058979 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/524dce36-43f9-45c7-bf24-6ad7918aeb94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:34.193167 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:34.193125 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:34.590548 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:34.590511 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:34.991934 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:34.991904 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:35.168884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:35.168848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:35.398956 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:35.398868 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:35.453560 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:35.453519 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:35.592642 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:35.592595 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:35.997081 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:35.997050 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:36.047479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:36.047450 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:36.403027 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:36.402936 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:36.996668 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:36.996632 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:37.097999 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:37.097964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:37.198524 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:37.198474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:37.594993 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:37.594956 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:37.695923 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:37.695887 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:37.796249 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:37.796211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:37.992648 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:37.992612 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:38.391366 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:38.391278 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:38.998005 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:38.997969 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:39.588601 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:39.588567 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:39.751068 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:39.751034 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:39.851432 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:39.851339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:40.198198 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:40.198108 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:40.400262 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:40.400227 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:40.730479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:40.730452 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xjmj" Jul 27 23:37:40.791265 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:40.791231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:41.190868 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:41.190827 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:41.795306 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:41.795256 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:41.989375 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:41.989334 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:42.121223 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.121133 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:42.123471 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.123449 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/524dce36-43f9-45c7-bf24-6ad7918aeb94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46fmv\" (UID: \"524dce36-43f9-45c7-bf24-6ad7918aeb94\") " pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:42.185033 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.185006 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sbzb5\"" Jul 27 23:37:42.193177 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.193137 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:42.193894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.193876 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-46fmv" Jul 27 23:37:42.315365 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.315285 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-46fmv"] Jul 27 23:37:42.317894 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:42.317861 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524dce36_43f9_45c7_bf24_6ad7918aeb94.slice/crio-272b7e001cbf48e8e78e03ef83e2e9fff8b5a96b92ba63f1d57c0ddc16a9c2bc WatchSource:0}: Error finding container 272b7e001cbf48e8e78e03ef83e2e9fff8b5a96b92ba63f1d57c0ddc16a9c2bc: Status 404 returned error can't find the container with id 272b7e001cbf48e8e78e03ef83e2e9fff8b5a96b92ba63f1d57c0ddc16a9c2bc Jul 27 23:37:42.732789 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.732752 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:42.815785 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.815705 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46fmv" event={"ID":"524dce36-43f9-45c7-bf24-6ad7918aeb94","Type":"ContainerStarted","Data":"272b7e001cbf48e8e78e03ef83e2e9fff8b5a96b92ba63f1d57c0ddc16a9c2bc"} Jul 27 23:37:42.833907 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:42.833829 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:45.879447 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.879411 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-796945c6c4-jrxlv"] Jul 27 23:37:45.879906 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:37:45.879653 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" podUID="47955614-5597-4835-9f17-48edb322ded4" Jul 27 23:37:45.913639 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.913607 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx"] Jul 27 23:37:45.934338 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.934074 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj"] Jul 27 23:37:45.934338 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.934196 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:45.936786 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.936725 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 23:37:45.937371 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.937227 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 23:37:45.937371 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.937329 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wf6l9\"" Jul 27 23:37:45.938046 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.938020 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 27 23:37:45.938274 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.938253 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk"] Jul 27 23:37:45.938415 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.938400 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:45.939445 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.938946 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 23:37:45.944568 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.944545 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 27 23:37:45.944743 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.944720 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:45.944862 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.944640 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx"] Jul 27 23:37:45.944862 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.944852 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk"] Jul 27 23:37:45.944958 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.944874 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj"] Jul 27 23:37:45.945708 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.945646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:45.946627 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.946607 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 23:37:45.946779 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.946761 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 23:37:45.946892 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.946869 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 23:37:45.947130 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.946996 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 23:37:45.953731 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.953680 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2wd\" (UniqueName: \"kubernetes.io/projected/499735f4-6c96-41c0-888f-5c566e1b7746-kube-api-access-mn2wd\") pod \"managed-serviceaccount-addon-agent-6765ff68b-tjbqx\" (UID: \"499735f4-6c96-41c0-888f-5c566e1b7746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:45.953860 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:45.953728 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/499735f4-6c96-41c0-888f-5c566e1b7746-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6765ff68b-tjbqx\" (UID: \"499735f4-6c96-41c0-888f-5c566e1b7746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:46.003538 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.003487 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bc54fbf6b-4qdsv"] Jul 27 23:37:46.007126 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.007100 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.016881 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.016854 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc54fbf6b-4qdsv"] Jul 27 23:37:46.030775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.030742 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:46.054185 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054132 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f6a536d-530c-481f-8db5-3718f487a32f-registry-certificates\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054185 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054172 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.054395 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054202 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrwc\" (UniqueName: \"kubernetes.io/projected/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-kube-api-access-kjrwc\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.054395 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054334 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f6a536d-530c-481f-8db5-3718f487a32f-image-registry-private-configuration\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054513 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054396 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.054513 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054433 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbe71c1-1256-4b06-9d77-8f618b42176e-tmp\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.054513 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054466 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f6a536d-530c-481f-8db5-3718f487a32f-ca-trust-extracted\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054515 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.054670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054546 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/499735f4-6c96-41c0-888f-5c566e1b7746-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6765ff68b-tjbqx\" (UID: \"499735f4-6c96-41c0-888f-5c566e1b7746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:46.054670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6js\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-kube-api-access-kt6js\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054630 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f6a536d-530c-481f-8db5-3718f487a32f-installation-pull-secrets\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054666 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a536d-530c-481f-8db5-3718f487a32f-trusted-ca\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054698 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2wd\" (UniqueName: \"kubernetes.io/projected/499735f4-6c96-41c0-888f-5c566e1b7746-kube-api-access-mn2wd\") pod \"managed-serviceaccount-addon-agent-6765ff68b-tjbqx\" (UID: \"499735f4-6c96-41c0-888f-5c566e1b7746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:46.054880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054770 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7bbe71c1-1256-4b06-9d77-8f618b42176e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.054880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054824 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-bound-sa-token\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.054880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054860 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-ca\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.055059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054889 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-registry-tls\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.055059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054924 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfj7\" (UniqueName: \"kubernetes.io/projected/7bbe71c1-1256-4b06-9d77-8f618b42176e-kube-api-access-wxfj7\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.055059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.054950 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-hub\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.057228 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.057205 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/499735f4-6c96-41c0-888f-5c566e1b7746-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6765ff68b-tjbqx\" (UID: \"499735f4-6c96-41c0-888f-5c566e1b7746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:46.063052 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.063031 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2wd\" (UniqueName: \"kubernetes.io/projected/499735f4-6c96-41c0-888f-5c566e1b7746-kube-api-access-mn2wd\") pod \"managed-serviceaccount-addon-agent-6765ff68b-tjbqx\" (UID: \"499735f4-6c96-41c0-888f-5c566e1b7746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:46.129339 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.129262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:46.155600 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155486 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f6a536d-530c-481f-8db5-3718f487a32f-installation-pull-secrets\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.155600 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155554 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a536d-530c-481f-8db5-3718f487a32f-trusted-ca\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.155600 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155601 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7bbe71c1-1256-4b06-9d77-8f618b42176e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-bound-sa-token\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-ca\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155713 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-registry-tls\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155742 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfj7\" (UniqueName: \"kubernetes.io/projected/7bbe71c1-1256-4b06-9d77-8f618b42176e-kube-api-access-wxfj7\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155764 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-hub\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f6a536d-530c-481f-8db5-3718f487a32f-registry-certificates\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155822 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.155884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155853 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrwc\" (UniqueName: \"kubernetes.io/projected/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-kube-api-access-kjrwc\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.156354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155923 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f6a536d-530c-481f-8db5-3718f487a32f-image-registry-private-configuration\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.156354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155955 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.156354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.155983 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbe71c1-1256-4b06-9d77-8f618b42176e-tmp\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.156354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.156010 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f6a536d-530c-481f-8db5-3718f487a32f-ca-trust-extracted\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.156354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.156037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.156354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.156067 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6js\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-kube-api-access-kt6js\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.156701 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.156627 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a536d-530c-481f-8db5-3718f487a32f-trusted-ca\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.158143 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.157044 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f6a536d-530c-481f-8db5-3718f487a32f-registry-certificates\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.158143 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.157368 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbe71c1-1256-4b06-9d77-8f618b42176e-tmp\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.158143 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.157648 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f6a536d-530c-481f-8db5-3718f487a32f-ca-trust-extracted\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.158143 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.158100 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.159300 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.159272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f6a536d-530c-481f-8db5-3718f487a32f-image-registry-private-configuration\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.160013 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.159990 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-registry-tls\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.160150 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.160125 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-hub\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.160456 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.160433 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-ca\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.160671 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.160649 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.161372 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.161347 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f6a536d-530c-481f-8db5-3718f487a32f-installation-pull-secrets\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.161635 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.161577 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.161950 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.161930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7bbe71c1-1256-4b06-9d77-8f618b42176e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.165596 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.165556 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-bound-sa-token\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.165848 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.165829 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfj7\" (UniqueName: \"kubernetes.io/projected/7bbe71c1-1256-4b06-9d77-8f618b42176e-kube-api-access-wxfj7\") pod \"klusterlet-addon-workmgr-5b7ddcb88-tmvbj\" (UID: \"7bbe71c1-1256-4b06-9d77-8f618b42176e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.166611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.166589 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6js\" (UniqueName: \"kubernetes.io/projected/0f6a536d-530c-481f-8db5-3718f487a32f-kube-api-access-kt6js\") pod \"image-registry-6bc54fbf6b-4qdsv\" (UID: \"0f6a536d-530c-481f-8db5-3718f487a32f\") " pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.166821 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.166799 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrwc\" (UniqueName: \"kubernetes.io/projected/4d4fa7f6-8eef-4ff1-920c-faf862b42aad-kube-api-access-kjrwc\") pod \"cluster-proxy-proxy-agent-6db55df9b6-9vzdk\" (UID: \"4d4fa7f6-8eef-4ff1-920c-faf862b42aad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.231932 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.231893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:46.257047 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.257016 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:46.257193 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.257084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:46.259125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.259103 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 23:37:46.259240 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.259210 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 23:37:46.259379 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.259364 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" Jul 27 23:37:46.268256 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.268235 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:46.269929 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.269909 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ab68dc8-3b95-4c0e-9a6c-611abfc63751-original-pull-secret\") pod \"global-pull-secret-syncer-gqttn\" (UID: \"5ab68dc8-3b95-4c0e-9a6c-611abfc63751\") " pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:46.270243 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.270229 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 23:37:46.275032 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.275012 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" Jul 27 23:37:46.280621 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.280601 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dpt\" (UniqueName: \"kubernetes.io/projected/faafc2d2-a65b-4cfd-990b-7da73a6454f4-kube-api-access-p7dpt\") pod \"network-check-target-89dpp\" (UID: \"faafc2d2-a65b-4cfd-990b-7da73a6454f4\") " pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:46.284481 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.284461 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqttn" Jul 27 23:37:46.334014 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.333987 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mkzjz\"" Jul 27 23:37:46.342890 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.342861 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:46.357732 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.357697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:46.359809 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.359786 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 23:37:46.370510 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.370472 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25d840-3aaa-417d-9ede-4a6e9f79d730-metrics-certs\") pod \"network-metrics-daemon-ppxkd\" (UID: \"3e25d840-3aaa-417d-9ede-4a6e9f79d730\") " pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:46.575245 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.575216 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8p7f9\"" Jul 27 23:37:46.580863 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.580839 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t92bh\"" Jul 27 23:37:46.583316 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.583298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:46.589787 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.589767 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ppxkd" Jul 27 23:37:46.827196 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.827080 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:46.835906 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.835872 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:46.862307 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862269 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-trusted-ca\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862452 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862333 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-registry-certificates\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862452 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862363 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpg4j\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-kube-api-access-wpg4j\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862452 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862417 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-image-registry-private-configuration\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862478 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-bound-sa-token\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862532 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47955614-5597-4835-9f17-48edb322ded4-ca-trust-extracted\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862566 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-installation-pull-secrets\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:46.862902 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862825 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47955614-5597-4835-9f17-48edb322ded4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:37:46.863029 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.862985 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:37:46.863234 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.863204 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:37:46.865254 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.865225 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:37:46.865347 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.865221 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-kube-api-access-wpg4j" (OuterVolumeSpecName: "kube-api-access-wpg4j") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "kube-api-access-wpg4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:37:46.865347 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.865284 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:37:46.865645 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.865609 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:37:46.963488 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963456 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-installation-pull-secrets\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:46.963488 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963488 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-trusted-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:46.963854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963513 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47955614-5597-4835-9f17-48edb322ded4-registry-certificates\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:46.963854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963522 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpg4j\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-kube-api-access-wpg4j\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:46.963854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963532 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47955614-5597-4835-9f17-48edb322ded4-image-registry-private-configuration\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:46.963854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963541 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-bound-sa-token\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:46.963854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:46.963549 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47955614-5597-4835-9f17-48edb322ded4-ca-trust-extracted\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:47.168103 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.167353 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx"] Jul 27 23:37:47.169767 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.168821 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj"] Jul 27 23:37:47.171255 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:47.171181 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod499735f4_6c96_41c0_888f_5c566e1b7746.slice/crio-3ecd2324a8c6ca529ee5251463367b361849803da574ff918d6642ebbad2a762 WatchSource:0}: Error finding container 3ecd2324a8c6ca529ee5251463367b361849803da574ff918d6642ebbad2a762: Status 404 returned error can't find the container with id 3ecd2324a8c6ca529ee5251463367b361849803da574ff918d6642ebbad2a762 Jul 27 23:37:47.177488 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:47.177447 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbe71c1_1256_4b06_9d77_8f618b42176e.slice/crio-7a5e697ccca59c2aa078f91368e0a55e81544838d3c7f4e6036fab80a288bd74 WatchSource:0}: Error finding container 7a5e697ccca59c2aa078f91368e0a55e81544838d3c7f4e6036fab80a288bd74: Status 404 returned error can't find the container with id 7a5e697ccca59c2aa078f91368e0a55e81544838d3c7f4e6036fab80a288bd74 Jul 27 23:37:47.229197 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.229167 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89dpp"] Jul 27 23:37:47.232187 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:47.232144 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaafc2d2_a65b_4cfd_990b_7da73a6454f4.slice/crio-097d3631454ee620c8a0c3f10e850708c37587c2bd40cc8d0487c05944e87c0d WatchSource:0}: Error finding container 097d3631454ee620c8a0c3f10e850708c37587c2bd40cc8d0487c05944e87c0d: Status 404 returned error can't find the container with id 097d3631454ee620c8a0c3f10e850708c37587c2bd40cc8d0487c05944e87c0d Jul 27 23:37:47.405705 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.405555 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ppxkd"] Jul 27 23:37:47.408330 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.406643 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqttn"] Jul 27 23:37:47.411212 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:47.411189 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab68dc8_3b95_4c0e_9a6c_611abfc63751.slice/crio-b6e0b45b55e6fa0b04be3c7fa99804f9e765268619d7fbdf4f7ba56107301b55 WatchSource:0}: Error finding container b6e0b45b55e6fa0b04be3c7fa99804f9e765268619d7fbdf4f7ba56107301b55: Status 404 returned error can't find the container with id b6e0b45b55e6fa0b04be3c7fa99804f9e765268619d7fbdf4f7ba56107301b55 Jul 27 23:37:47.453437 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.453388 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc54fbf6b-4qdsv"] Jul 27 23:37:47.454371 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.454345 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk"] Jul 27 23:37:47.456650 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:47.456622 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4fa7f6_8eef_4ff1_920c_faf862b42aad.slice/crio-2727245e43648ea1a4f2a12929441cbf1f9c8f4620d8566d36c9660966ff4308 WatchSource:0}: Error finding container 2727245e43648ea1a4f2a12929441cbf1f9c8f4620d8566d36c9660966ff4308: Status 404 returned error can't find the container with id 2727245e43648ea1a4f2a12929441cbf1f9c8f4620d8566d36c9660966ff4308 Jul 27 23:37:47.457174 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:47.457159 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6a536d_530c_481f_8db5_3718f487a32f.slice/crio-3573c9113ee00256506bda2695d1f07c02f900bcdb3d83645a43c68424f86d92 WatchSource:0}: Error finding container 3573c9113ee00256506bda2695d1f07c02f900bcdb3d83645a43c68424f86d92: Status 404 returned error can't find the container with id 3573c9113ee00256506bda2695d1f07c02f900bcdb3d83645a43c68424f86d92 Jul 27 23:37:47.771975 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.771414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:47.771975 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.771486 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:47.774543 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.774512 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"image-registry-796945c6c4-jrxlv\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:47.775349 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.775311 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b59dc8-4046-499e-97e0-3192fa260310-metrics-tls\") pod \"dns-default-lqr4s\" (UID: \"65b59dc8-4046-499e-97e0-3192fa260310\") " pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:47.834271 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.834236 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqttn" event={"ID":"5ab68dc8-3b95-4c0e-9a6c-611abfc63751","Type":"ContainerStarted","Data":"b6e0b45b55e6fa0b04be3c7fa99804f9e765268619d7fbdf4f7ba56107301b55"} Jul 27 23:37:47.835773 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.835733 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" event={"ID":"7bbe71c1-1256-4b06-9d77-8f618b42176e","Type":"ContainerStarted","Data":"7a5e697ccca59c2aa078f91368e0a55e81544838d3c7f4e6036fab80a288bd74"} Jul 27 23:37:47.837957 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.837917 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" event={"ID":"499735f4-6c96-41c0-888f-5c566e1b7746","Type":"ContainerStarted","Data":"3ecd2324a8c6ca529ee5251463367b361849803da574ff918d6642ebbad2a762"} Jul 27 23:37:47.840479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.840456 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" event={"ID":"0f6a536d-530c-481f-8db5-3718f487a32f","Type":"ContainerStarted","Data":"9ef80c9511858baa7bc9c497c504f0edd2e1133b78956f3fdb01574a75b16d04"} Jul 27 23:37:47.840664 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.840544 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" event={"ID":"0f6a536d-530c-481f-8db5-3718f487a32f","Type":"ContainerStarted","Data":"3573c9113ee00256506bda2695d1f07c02f900bcdb3d83645a43c68424f86d92"} Jul 27 23:37:47.840664 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.840600 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:37:47.842302 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.842247 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89dpp" event={"ID":"faafc2d2-a65b-4cfd-990b-7da73a6454f4","Type":"ContainerStarted","Data":"097d3631454ee620c8a0c3f10e850708c37587c2bd40cc8d0487c05944e87c0d"} Jul 27 23:37:47.844112 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.844076 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" event={"ID":"4d4fa7f6-8eef-4ff1-920c-faf862b42aad","Type":"ContainerStarted","Data":"2727245e43648ea1a4f2a12929441cbf1f9c8f4620d8566d36c9660966ff4308"} Jul 27 23:37:47.845750 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.845728 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ppxkd" event={"ID":"3e25d840-3aaa-417d-9ede-4a6e9f79d730","Type":"ContainerStarted","Data":"5a6684c5f501d16a8ff53e6be46ad8bdc16131cba2713d236bdcc79b030d3000"} Jul 27 23:37:47.848419 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.848398 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796945c6c4-jrxlv" Jul 27 23:37:47.848602 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.848545 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46fmv" event={"ID":"524dce36-43f9-45c7-bf24-6ad7918aeb94","Type":"ContainerStarted","Data":"155b12299e16c21a2d15c69e764178870a006df9d43470ce2bf841bb7976a57f"} Jul 27 23:37:47.858398 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.858172 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" podStartSLOduration=2.85815513 podStartE2EDuration="2.85815513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:47.857411423 +0000 UTC m=+67.781620302" watchObservedRunningTime="2026-07-27 23:37:47.85815513 +0000 UTC m=+67.782364015" Jul 27 23:37:47.871900 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.871861 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") pod \"47955614-5597-4835-9f17-48edb322ded4\" (UID: \"47955614-5597-4835-9f17-48edb322ded4\") " Jul 27 23:37:47.874340 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.874303 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "47955614-5597-4835-9f17-48edb322ded4" (UID: "47955614-5597-4835-9f17-48edb322ded4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:37:47.973122 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:47.973078 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47955614-5597-4835-9f17-48edb322ded4-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:37:48.023110 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.022994 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jk8w\"" Jul 27 23:37:48.031262 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.030983 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:48.196132 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.196069 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-796945c6c4-jrxlv"] Jul 27 23:37:48.208011 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.207730 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-796945c6c4-jrxlv"] Jul 27 23:37:48.248339 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.248261 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqr4s"] Jul 27 23:37:48.250181 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.249823 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:48.257519 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:37:48.257462 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b59dc8_4046_499e_97e0_3192fa260310.slice/crio-6d4b018613ceab8cd17f90f3355c5f199d87312531e8de7d3e1ee036787ab85a WatchSource:0}: Error finding container 6d4b018613ceab8cd17f90f3355c5f199d87312531e8de7d3e1ee036787ab85a: Status 404 returned error can't find the container with id 6d4b018613ceab8cd17f90f3355c5f199d87312531e8de7d3e1ee036787ab85a Jul 27 23:37:48.343565 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.343426 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:48.402365 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.402043 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:48.543704 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.543641 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:48.566444 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.566223 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47955614-5597-4835-9f17-48edb322ded4" path="/var/lib/kubelet/pods/47955614-5597-4835-9f17-48edb322ded4/volumes" Jul 27 23:37:48.716812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.716673 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:48.772266 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.772157 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:48.877224 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.877142 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqr4s" event={"ID":"65b59dc8-4046-499e-97e0-3192fa260310","Type":"ContainerStarted","Data":"6d4b018613ceab8cd17f90f3355c5f199d87312531e8de7d3e1ee036787ab85a"} Jul 27 23:37:48.952721 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:48.952686 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:49.008273 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:49.008209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:49.548527 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:49.544835 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:49.596080 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:49.595984 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:49.921292 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:49.921185 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:50.020359 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:50.020273 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:50.123943 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:50.123873 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:50.317319 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:50.317236 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:50.894529 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:50.894475 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ppxkd" event={"ID":"3e25d840-3aaa-417d-9ede-4a6e9f79d730","Type":"ContainerStarted","Data":"c25fbfbc95b12dba1bdc8c2894b18418cf8a3bbc5e7502551b2040e29b212223"} Jul 27 23:37:50.894720 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:50.894563 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ppxkd" event={"ID":"3e25d840-3aaa-417d-9ede-4a6e9f79d730","Type":"ContainerStarted","Data":"d367b2cce017ef953f288fd0f80803eb557dd245849c8855212c2d3bee49a7e9"} Jul 27 23:37:50.911184 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:50.911125 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ppxkd" podStartSLOduration=68.217386937 podStartE2EDuration="1m10.911106099s" totalImagesPullingTime="2.693719162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:50.910601456 +0000 UTC m=+70.834810349" watchObservedRunningTime="2026-07-27 23:37:50.911106099 +0000 UTC m=+70.835314978" Jul 27 23:37:51.028254 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:51.028186 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:51.086424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:51.086361 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:51.733728 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:51.733686 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:51.791099 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:51.790976 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:52.140986 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:52.140891 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:57.576765 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:57.576701 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:57.675072 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:57.675033 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:57.761706 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:57.761666 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:57.820438 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:57.820401 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:57.920928 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:57.920848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:57.973175 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:57.973139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:37:59.927843 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.927803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89dpp" event={"ID":"faafc2d2-a65b-4cfd-990b-7da73a6454f4","Type":"ContainerStarted","Data":"cab1012a301856ca557ea3c1700601407bc0eb4c5c3ecc36984f7e13a6d4210e"} Jul 27 23:37:59.928292 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.928000 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:37:59.929983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.929951 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" event={"ID":"4d4fa7f6-8eef-4ff1-920c-faf862b42aad","Type":"ContainerStarted","Data":"6529cc084b4799be6c9a5dd43c5ca25e5760f29622e0b53f40b0164937343ec0"} Jul 27 23:37:59.930105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.929990 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" event={"ID":"4d4fa7f6-8eef-4ff1-920c-faf862b42aad","Type":"ContainerStarted","Data":"cd7dafd251e9fea43496de8babd867df0d8fe0d2091147154842e5ca98a0edd8"} Jul 27 23:37:59.930105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.930007 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" event={"ID":"4d4fa7f6-8eef-4ff1-920c-faf862b42aad","Type":"ContainerStarted","Data":"afcb302b90adbb924cd1120766e49e5ebb4be1f417e39b147e61f3418b6a4bce"} Jul 27 23:37:59.931796 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.931774 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46fmv" event={"ID":"524dce36-43f9-45c7-bf24-6ad7918aeb94","Type":"ContainerStarted","Data":"1f9ac81e374747606ab7170fa065d8d2a2b1351192a9639cc5ae7a56eac0cfab"} Jul 27 23:37:59.933128 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.933105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqttn" event={"ID":"5ab68dc8-3b95-4c0e-9a6c-611abfc63751","Type":"ContainerStarted","Data":"37d774d23baa578e052a141dc13e41da5b28bb662637b7b07d9b996630142536"} Jul 27 23:37:59.934534 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.934512 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" event={"ID":"7bbe71c1-1256-4b06-9d77-8f618b42176e","Type":"ContainerStarted","Data":"5b45ee05635bb090198d6fb606d3431585610cbeaae8c39b2121cc7ebbb8fd97"} Jul 27 23:37:59.934617 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.934552 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:59.936160 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.936137 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" event={"ID":"499735f4-6c96-41c0-888f-5c566e1b7746","Type":"ContainerStarted","Data":"4e81207ac3995d698ed29c8ed07c297e1dc13b0b86e9f278329cd878cbb4fe81"} Jul 27 23:37:59.936387 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.936371 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" Jul 27 23:37:59.937948 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.937927 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqr4s" event={"ID":"65b59dc8-4046-499e-97e0-3192fa260310","Type":"ContainerStarted","Data":"1db3fe34f6a4ecacbe43f0820670800f73a73a9d2f85ba0daf1e630688f2e74b"} Jul 27 23:37:59.938034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.937961 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqr4s" event={"ID":"65b59dc8-4046-499e-97e0-3192fa260310","Type":"ContainerStarted","Data":"e533a21ea8b6812f589172372940b02fa57edb7cf3df985a47880a77e699b728"} Jul 27 23:37:59.938111 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.938058 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lqr4s" Jul 27 23:37:59.942392 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.942351 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-89dpp" podStartSLOduration=67.688702487 podStartE2EDuration="1m19.942339734s" totalImagesPullingTime="12.253637247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:59.94107176 +0000 UTC m=+79.865280638" watchObservedRunningTime="2026-07-27 23:37:59.942339734 +0000 UTC m=+79.866548612" Jul 27 23:37:59.955921 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.955883 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b7ddcb88-tmvbj" podStartSLOduration=2.567490577 podStartE2EDuration="14.95587099s" totalImagesPullingTime="12.388380413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:59.954883256 +0000 UTC m=+79.879092134" watchObservedRunningTime="2026-07-27 23:37:59.95587099 +0000 UTC m=+79.880079885" Jul 27 23:37:59.972056 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.972018 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db55df9b6-9vzdk" podStartSLOduration=2.862000902 podStartE2EDuration="14.972007131s" totalImagesPullingTime="12.110006229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:59.971581231 +0000 UTC m=+79.895790108" watchObservedRunningTime="2026-07-27 23:37:59.972007131 +0000 UTC m=+79.896216009" Jul 27 23:37:59.984503 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.984464 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6765ff68b-tjbqx" podStartSLOduration=2.670274213 podStartE2EDuration="14.984452769s" totalImagesPullingTime="12.314178556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:59.983707874 +0000 UTC m=+79.907916753" watchObservedRunningTime="2026-07-27 23:37:59.984452769 +0000 UTC m=+79.908661647" Jul 27 23:37:59.999482 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:37:59.999413 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gqttn" podStartSLOduration=67.843440354 podStartE2EDuration="1m19.99940247s" totalImagesPullingTime="12.155962116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:37:59.998861846 +0000 UTC m=+79.923070726" watchObservedRunningTime="2026-07-27 23:37:59.99940247 +0000 UTC m=+79.923611325" Jul 27 23:38:00.019014 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:00.018976 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lqr4s" podStartSLOduration=33.7109647 podStartE2EDuration="45.018959883s" totalImagesPullingTime="11.307995183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:38:00.018229858 +0000 UTC m=+79.942438735" watchObservedRunningTime="2026-07-27 23:38:00.018959883 +0000 UTC m=+79.943168761" Jul 27 23:38:00.548096 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:00.548047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:02.353682 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:02.353639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:02.456014 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:02.455975 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:03.789815 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:03.789777 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:03.861669 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:03.861631 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:03.954063 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:03.954022 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46fmv" event={"ID":"524dce36-43f9-45c7-bf24-6ad7918aeb94","Type":"ContainerStarted","Data":"3267baae9f361f97ae6579b0a3260810fb0cfe088bbb6fdf3d55e1e1a46df947"} Jul 27 23:38:03.969611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:03.969561 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-46fmv" podStartSLOduration=17.408669374 podStartE2EDuration="37.969546639s" totalImagesPullingTime="20.560877265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:37:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:38:03.968738169 +0000 UTC m=+83.892947052" watchObservedRunningTime="2026-07-27 23:38:03.969546639 +0000 UTC m=+83.893755517" Jul 27 23:38:04.331596 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:04.331562 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:04.382956 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:04.382920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:06.347520 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:06.347471 2555 patch_prober.go:28] interesting pod/image-registry-6bc54fbf6b-4qdsv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 27 23:38:06.347906 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:06.347540 2555 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" podUID="0f6a536d-530c-481f-8db5-3718f487a32f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 23:38:06.520698 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:06.520664 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:07.514849 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:07.514783 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:07.615556 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:07.615517 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:07.933224 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:07.933138 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.147659 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.147620 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.200255 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.200176 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pcxdc"] Jul 27 23:38:08.237439 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.237400 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.379172 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.379138 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.381188 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.381163 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 23:38:08.381328 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.381221 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 23:38:08.381946 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.381927 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 23:38:08.382092 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.381970 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 23:38:08.382158 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.381973 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7v7md\"" Jul 27 23:38:08.382212 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.381988 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 23:38:08.382212 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.382016 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 23:38:08.437525 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-tls\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437686 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437534 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-sys\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437686 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437556 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-root\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437686 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437633 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0465816e-82d7-4e52-8780-9e38483682d7-metrics-client-ca\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437836 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437707 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-textfile\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437836 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437736 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82whx\" (UniqueName: \"kubernetes.io/projected/0465816e-82d7-4e52-8780-9e38483682d7-kube-api-access-82whx\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437836 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437767 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-wtmp\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437836 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437810 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.437836 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.437832 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.444878 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.444854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.538957 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.538918 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.538973 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-tls\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539011 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-sys\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539040 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-root\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539071 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0465816e-82d7-4e52-8780-9e38483682d7-metrics-client-ca\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:38:08.539094 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539123 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-textfile\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539153 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82whx\" (UniqueName: \"kubernetes.io/projected/0465816e-82d7-4e52-8780-9e38483682d7-kube-api-access-82whx\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539161 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-sys\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:38:08.539168 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-tls podName:0465816e-82d7-4e52-8780-9e38483682d7 nodeName:}" failed. No retries permitted until 2026-07-27 23:38:09.03914788 +0000 UTC m=+88.963356743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-tls") pod "node-exporter-pcxdc" (UID: "0465816e-82d7-4e52-8780-9e38483682d7") : secret "node-exporter-tls" not found Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539243 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-wtmp\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.539427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539346 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-root\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.540125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539577 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-wtmp\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.540125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539892 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-textfile\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.540125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539894 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.540125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.539960 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0465816e-82d7-4e52-8780-9e38483682d7-metrics-client-ca\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.540784 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.540759 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.541869 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.541850 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.552664 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.552643 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82whx\" (UniqueName: \"kubernetes.io/projected/0465816e-82d7-4e52-8780-9e38483682d7-kube-api-access-82whx\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:08.608386 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.608349 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.735658 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.735599 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.835668 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.835573 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:08.881453 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:08.881419 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bc54fbf6b-4qdsv" Jul 27 23:38:09.036706 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.036674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:09.044838 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.044810 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-tls\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:09.050574 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.050546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0465816e-82d7-4e52-8780-9e38483682d7-node-exporter-tls\") pod \"node-exporter-pcxdc\" (UID: \"0465816e-82d7-4e52-8780-9e38483682d7\") " pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:09.137982 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.137896 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:09.238450 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.238417 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:09.288551 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.288511 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcxdc" Jul 27 23:38:09.337880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.337822 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:09.344041 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:38:09.344012 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0465816e_82d7_4e52_8780_9e38483682d7.slice/crio-7aa4039664e42b2bd77290279ad8fff11d75dd9f8bb72fbd09a5e142cfe704af WatchSource:0}: Error finding container 7aa4039664e42b2bd77290279ad8fff11d75dd9f8bb72fbd09a5e142cfe704af: Status 404 returned error can't find the container with id 7aa4039664e42b2bd77290279ad8fff11d75dd9f8bb72fbd09a5e142cfe704af Jul 27 23:38:09.437755 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.437672 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:09.538889 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.538849 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:09.944721 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.944689 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lqr4s" Jul 27 23:38:09.976924 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:09.976890 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcxdc" event={"ID":"0465816e-82d7-4e52-8780-9e38483682d7","Type":"ContainerStarted","Data":"7aa4039664e42b2bd77290279ad8fff11d75dd9f8bb72fbd09a5e142cfe704af"} Jul 27 23:38:10.982044 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:10.981967 2555 generic.go:356] "Generic (PLEG): container finished" podID="0465816e-82d7-4e52-8780-9e38483682d7" containerID="e2a0f87da5fa18a15ef80093bbdde6d671804c7e2c689007237d17f54bcc7b73" exitCode=0 Jul 27 23:38:10.982383 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:10.982046 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcxdc" event={"ID":"0465816e-82d7-4e52-8780-9e38483682d7","Type":"ContainerDied","Data":"e2a0f87da5fa18a15ef80093bbdde6d671804c7e2c689007237d17f54bcc7b73"} Jul 27 23:38:11.133895 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:11.133855 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:11.237262 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:11.237166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:11.309346 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:11.309306 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:11.753514 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:11.753460 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:11.987424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:11.987390 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcxdc" event={"ID":"0465816e-82d7-4e52-8780-9e38483682d7","Type":"ContainerStarted","Data":"fdfc38680819ba5559ec321b525668f03922b56c3f77e2d503eaaa4356346346"} Jul 27 23:38:11.987823 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:11.987443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcxdc" event={"ID":"0465816e-82d7-4e52-8780-9e38483682d7","Type":"ContainerStarted","Data":"afa48d6b3fa76978eb9067bc7a4344e8770cf4d9600520b6cd8ca8c087453edc"} Jul 27 23:38:12.004683 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:12.004590 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pcxdc" podStartSLOduration=2.529724894 podStartE2EDuration="4.004574509s" totalImagesPullingTime="1.376204779s" totalInitContainerRuntime="98.644836ms" isStatefulPod=true podCreationTimestamp="2026-07-27 23:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:38:12.003898148 +0000 UTC m=+91.928107025" watchObservedRunningTime="2026-07-27 23:38:12.004574509 +0000 UTC m=+91.928783388" Jul 27 23:38:12.727241 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:12.727201 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:12.828444 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:12.828405 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:12.973074 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:12.973038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:13.039725 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:13.039690 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:13.298717 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:13.298624 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:13.719961 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:13.719869 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:13.775773 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:13.775734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:14.452415 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:14.452374 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:16.633886 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:16.633847 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:16.689362 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:16.689325 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:19.109346 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:19.109304 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:19.160861 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:19.160823 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:20.710814 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:20.710776 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:20.812722 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:20.812681 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:21.540434 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:21.540397 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:21.593069 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:21.593030 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:22.423651 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:22.423595 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:22.477720 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:22.477671 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:23.959823 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:23.959780 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:24.061696 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:24.061662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:24.971698 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:24.971654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:25.072750 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:25.072712 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:25.255397 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:25.255364 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:25.355749 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:25.355717 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:25.681285 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:25.681183 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:25.781608 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:25.781578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:26.573254 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:26.573220 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:26.673641 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:26.673610 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:29.848248 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:29.848218 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:29.946621 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:29.946583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:30.945531 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:30.945484 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-89dpp" Jul 27 23:38:35.472906 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:35.472866 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2fvf_a89c6a8f-50c8-46bb-ab07-ac6663f43fd9/serve-healthcheck-canary/0.log" Jul 27 23:38:42.737687 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:42.737654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:42.837553 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:42.837518 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:44.077069 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:44.077028 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:44.132302 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:44.132265 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:49.896349 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:49.896304 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:49.997464 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:49.997428 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:50.059095 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:50.059056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:57.965418 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:38:57.965189 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:10.152316 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:10.152267 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:27.460164 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:27.460124 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:27.764440 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:27.764409 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:37.826096 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:37.826017 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:37.883888 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:37.883835 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:42.708571 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:42.708526 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:42.765660 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:42.765632 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:43.172364 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:43.172335 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:43.234621 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:43.234595 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:43.777092 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:43.777049 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:54.264618 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:54.264578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:54.366525 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:54.366475 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:54.427727 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:54.427697 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:54.725601 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:54.725513 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:54.782578 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:54.782548 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:57.744041 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:57.743998 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:57.844201 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:39:57.844161 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:40:05.750585 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:40:05.750551 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:40:05.808777 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:40:05.808739 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:40:39.410028 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:40:39.409995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:40:39.506222 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:40:39.506189 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:00.977554 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:00.977516 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:01.750178 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:01.750144 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:04.592155 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:04.592076 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:04.692069 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:04.692031 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:06.147457 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:06.147421 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:06.245775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:06.245741 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:06.649277 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:06.649236 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:06.776679 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:06.776650 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:08.560245 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:08.560209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:08.760854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:08.760819 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:11.379857 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:11.379823 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:11.437267 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:11.437238 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:15.784108 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:15.784068 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:24.581385 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:24.581343 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:24.715280 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:24.715246 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:31.359029 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:31.358986 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:31.458724 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:31.458688 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:33.164131 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:33.164093 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:35.534646 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:35.534609 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:35.599051 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:35.599015 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:40.438742 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:40.438715 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 23:41:45.315669 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:45.315638 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:46.735849 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:46.735811 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:47.697598 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:47.697556 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:47.802196 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:47.802160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:48.312110 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.312077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:48.730790 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.730708 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:48.744738 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.744704 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv"] Jul 27 23:41:48.746796 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.746782 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.748984 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.748966 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 23:41:48.749329 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.749313 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rhhx7\"" Jul 27 23:41:48.749635 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.749613 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 23:41:48.749736 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.749616 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 23:41:48.749736 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.749654 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 23:41:48.776545 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.776520 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv"] Jul 27 23:41:48.859589 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.859552 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c53a53de-9d1d-4e58-b38b-e930aa3833fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.859589 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.859593 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c53a53de-9d1d-4e58-b38b-e930aa3833fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.860047 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.859615 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbkn\" (UniqueName: \"kubernetes.io/projected/c53a53de-9d1d-4e58-b38b-e930aa3833fb-kube-api-access-rkbkn\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.860047 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.859763 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c53a53de-9d1d-4e58-b38b-e930aa3833fb-tmp\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.860047 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.859824 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c53a53de-9d1d-4e58-b38b-e930aa3833fb-manifests\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.897308 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.897275 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:48.960777 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.960749 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c53a53de-9d1d-4e58-b38b-e930aa3833fb-manifests\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.960941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.960782 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c53a53de-9d1d-4e58-b38b-e930aa3833fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.960941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.960809 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c53a53de-9d1d-4e58-b38b-e930aa3833fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.960941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.960838 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbkn\" (UniqueName: \"kubernetes.io/projected/c53a53de-9d1d-4e58-b38b-e930aa3833fb-kube-api-access-rkbkn\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.960941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.960893 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c53a53de-9d1d-4e58-b38b-e930aa3833fb-tmp\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.961226 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.961204 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c53a53de-9d1d-4e58-b38b-e930aa3833fb-manifests\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.963240 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.963212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c53a53de-9d1d-4e58-b38b-e930aa3833fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.963330 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.963213 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c53a53de-9d1d-4e58-b38b-e930aa3833fb-tmp\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.963429 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.963413 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c53a53de-9d1d-4e58-b38b-e930aa3833fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.968837 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.968812 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbkn\" (UniqueName: \"kubernetes.io/projected/c53a53de-9d1d-4e58-b38b-e930aa3833fb-kube-api-access-rkbkn\") pod \"opendatahub-operator-controller-manager-79b58bb86f-s8hcv\" (UID: \"c53a53de-9d1d-4e58-b38b-e930aa3833fb\") " pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:48.977570 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:48.977547 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:49.056593 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.056561 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:49.079401 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.079373 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:49.187074 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.187037 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:49.206157 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.206126 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv"] Jul 27 23:41:49.212226 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:41:49.212201 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53a53de_9d1d_4e58_b38b_e930aa3833fb.slice/crio-4500bb913d34411ea38985c9c43b441bcfcd97c6805f07bf87ba546ee0b9fca8 WatchSource:0}: Error finding container 4500bb913d34411ea38985c9c43b441bcfcd97c6805f07bf87ba546ee0b9fca8: Status 404 returned error can't find the container with id 4500bb913d34411ea38985c9c43b441bcfcd97c6805f07bf87ba546ee0b9fca8 Jul 27 23:41:49.214632 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.214615 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 23:41:49.284107 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.284073 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:49.384842 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.384756 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:41:49.683885 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:49.683803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" event={"ID":"c53a53de-9d1d-4e58-b38b-e930aa3833fb","Type":"ContainerStarted","Data":"4500bb913d34411ea38985c9c43b441bcfcd97c6805f07bf87ba546ee0b9fca8"} Jul 27 23:41:52.695839 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:52.695807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" event={"ID":"c53a53de-9d1d-4e58-b38b-e930aa3833fb","Type":"ContainerStarted","Data":"559992eb169c0a531f84e47eab4819010632216d25aed7886dc007857905da5d"} Jul 27 23:41:53.701342 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:53.701312 2555 generic.go:356] "Generic (PLEG): container finished" podID="c53a53de-9d1d-4e58-b38b-e930aa3833fb" containerID="559992eb169c0a531f84e47eab4819010632216d25aed7886dc007857905da5d" exitCode=0 Jul 27 23:41:53.701809 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:53.701366 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" event={"ID":"c53a53de-9d1d-4e58-b38b-e930aa3833fb","Type":"ContainerDied","Data":"559992eb169c0a531f84e47eab4819010632216d25aed7886dc007857905da5d"} Jul 27 23:41:54.706882 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:54.706848 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" event={"ID":"c53a53de-9d1d-4e58-b38b-e930aa3833fb","Type":"ContainerStarted","Data":"a463e6367cd57329bd1cb0c983ed1b167fdc1abf29f81b366caf32f4b29d9943"} Jul 27 23:41:54.707380 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:54.706982 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:41:54.726831 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:41:54.726777 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" podStartSLOduration=2.481416292 podStartE2EDuration="6.726751597s" totalImagesPullingTime="3.207381682s" totalInitContainerRuntime="1.037953623s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:41:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:41:54.725424695 +0000 UTC m=+314.649633573" watchObservedRunningTime="2026-07-27 23:41:54.726751597 +0000 UTC m=+314.650960475" Jul 27 23:42:05.712223 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:05.712193 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79b58bb86f-s8hcv" Jul 27 23:42:07.414639 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:07.414600 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:07.524639 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:07.524606 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.118963 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.118926 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.280855 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.280818 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw"] Jul 27 23:42:09.286316 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.286295 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.288912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.288890 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 23:42:09.289024 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.288933 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 23:42:09.289184 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.289171 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 23:42:09.289671 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.289656 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jcb5t\"" Jul 27 23:42:09.291423 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.291399 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 23:42:09.301569 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.301544 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw"] Jul 27 23:42:09.373934 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.373850 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.411715 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.411685 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kqc\" (UniqueName: \"kubernetes.io/projected/c8bc1753-f028-4a6a-97d1-021a0aad7b74-kube-api-access-t7kqc\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.411880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.411735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8bc1753-f028-4a6a-97d1-021a0aad7b74-tmp\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.411880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.411800 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bc1753-f028-4a6a-97d1-021a0aad7b74-tls-certs\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.432893 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.432861 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pnshl"] Jul 27 23:42:09.435032 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.435017 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.437290 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.437239 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 23:42:09.437762 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.437747 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rn2n6\"" Jul 27 23:42:09.445182 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.445157 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pnshl"] Jul 27 23:42:09.472281 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.472250 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.512923 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.512888 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8bc1753-f028-4a6a-97d1-021a0aad7b74-tmp\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.513065 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.512944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bc1753-f028-4a6a-97d1-021a0aad7b74-tls-certs\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.513065 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.513012 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kqc\" (UniqueName: \"kubernetes.io/projected/c8bc1753-f028-4a6a-97d1-021a0aad7b74-kube-api-access-t7kqc\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.515136 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.515105 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8bc1753-f028-4a6a-97d1-021a0aad7b74-tmp\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.515360 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.515340 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bc1753-f028-4a6a-97d1-021a0aad7b74-tls-certs\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.520573 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.520550 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kqc\" (UniqueName: \"kubernetes.io/projected/c8bc1753-f028-4a6a-97d1-021a0aad7b74-kube-api-access-t7kqc\") pod \"kube-auth-proxy-6f44c56477-pjfvw\" (UID: \"c8bc1753-f028-4a6a-97d1-021a0aad7b74\") " pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.570072 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.570040 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.595880 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.595859 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" Jul 27 23:42:09.613760 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.613737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8b6038f-3e24-4d26-87cd-215ee044673c-cert\") pod \"odh-model-controller-7998c67dc4-pnshl\" (UID: \"e8b6038f-3e24-4d26-87cd-215ee044673c\") " pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.613875 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.613805 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j27z\" (UniqueName: \"kubernetes.io/projected/e8b6038f-3e24-4d26-87cd-215ee044673c-kube-api-access-9j27z\") pod \"odh-model-controller-7998c67dc4-pnshl\" (UID: \"e8b6038f-3e24-4d26-87cd-215ee044673c\") " pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.714382 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.714347 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8b6038f-3e24-4d26-87cd-215ee044673c-cert\") pod \"odh-model-controller-7998c67dc4-pnshl\" (UID: \"e8b6038f-3e24-4d26-87cd-215ee044673c\") " pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.714539 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.714418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j27z\" (UniqueName: \"kubernetes.io/projected/e8b6038f-3e24-4d26-87cd-215ee044673c-kube-api-access-9j27z\") pod \"odh-model-controller-7998c67dc4-pnshl\" (UID: \"e8b6038f-3e24-4d26-87cd-215ee044673c\") " pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.716710 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.716689 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8b6038f-3e24-4d26-87cd-215ee044673c-cert\") pod \"odh-model-controller-7998c67dc4-pnshl\" (UID: \"e8b6038f-3e24-4d26-87cd-215ee044673c\") " pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.726336 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.726309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j27z\" (UniqueName: \"kubernetes.io/projected/e8b6038f-3e24-4d26-87cd-215ee044673c-kube-api-access-9j27z\") pod \"odh-model-controller-7998c67dc4-pnshl\" (UID: \"e8b6038f-3e24-4d26-87cd-215ee044673c\") " pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.744191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.744171 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:09.746008 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.745987 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw"] Jul 27 23:42:09.748805 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:42:09.748783 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bc1753_f028_4a6a_97d1_021a0aad7b74.slice/crio-f54dc7b6d5812bd6eaf2847a586c579e15615d70a37c857e132e8260bb5a7a24 WatchSource:0}: Error finding container f54dc7b6d5812bd6eaf2847a586c579e15615d70a37c857e132e8260bb5a7a24: Status 404 returned error can't find the container with id f54dc7b6d5812bd6eaf2847a586c579e15615d70a37c857e132e8260bb5a7a24 Jul 27 23:42:09.752181 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.752154 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.759744 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.759722 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" event={"ID":"c8bc1753-f028-4a6a-97d1-021a0aad7b74","Type":"ContainerStarted","Data":"f54dc7b6d5812bd6eaf2847a586c579e15615d70a37c857e132e8260bb5a7a24"} Jul 27 23:42:09.831341 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.831281 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:09.870684 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:09.870532 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pnshl"] Jul 27 23:42:09.873158 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:42:09.873132 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b6038f_3e24_4d26_87cd_215ee044673c.slice/crio-642128045b91f1a1411fa4a10b9be398616cf9d9cc45d6b99a954c361ae1654b WatchSource:0}: Error finding container 642128045b91f1a1411fa4a10b9be398616cf9d9cc45d6b99a954c361ae1654b: Status 404 returned error can't find the container with id 642128045b91f1a1411fa4a10b9be398616cf9d9cc45d6b99a954c361ae1654b Jul 27 23:42:10.126962 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.126923 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:10.127517 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.127481 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6dfjv"] Jul 27 23:42:10.130652 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.130631 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.132840 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.132820 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 27 23:42:10.133241 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.133224 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9gzd2\"" Jul 27 23:42:10.149141 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.149121 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6dfjv"] Jul 27 23:42:10.320228 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.320193 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec-tls-certs\") pod \"model-serving-api-54d89bd5fc-6dfjv\" (UID: \"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.320228 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.320227 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89d8\" (UniqueName: \"kubernetes.io/projected/2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec-kube-api-access-m89d8\") pod \"model-serving-api-54d89bd5fc-6dfjv\" (UID: \"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.404000 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.403925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:10.420933 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.420901 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec-tls-certs\") pod \"model-serving-api-54d89bd5fc-6dfjv\" (UID: \"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.421228 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.420947 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89d8\" (UniqueName: \"kubernetes.io/projected/2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec-kube-api-access-m89d8\") pod \"model-serving-api-54d89bd5fc-6dfjv\" (UID: \"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.427036 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.426979 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec-tls-certs\") pod \"model-serving-api-54d89bd5fc-6dfjv\" (UID: \"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.430729 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.430560 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89d8\" (UniqueName: \"kubernetes.io/projected/2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec-kube-api-access-m89d8\") pod \"model-serving-api-54d89bd5fc-6dfjv\" (UID: \"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.440885 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.440646 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:10.474396 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.474251 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms"] Jul 27 23:42:10.477299 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.477267 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:10.477424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.477277 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.479982 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.479875 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 23:42:10.479982 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.479966 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cj4hx\"" Jul 27 23:42:10.480158 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.479996 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 23:42:10.480158 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.480123 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 23:42:10.480270 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.480209 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 23:42:10.497656 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.497157 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms"] Jul 27 23:42:10.537866 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.537792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:10.618827 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.618789 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6dfjv"] Jul 27 23:42:10.622116 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622087 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.622201 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622163 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvck\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-kube-api-access-mnvck\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.622242 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622205 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.622276 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622238 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee25149-b18f-43d8-ab4e-e9d903669f1c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.622309 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622285 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.622341 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622318 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.622374 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.622346 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723307 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvck\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-kube-api-access-mnvck\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723360 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723397 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee25149-b18f-43d8-ab4e-e9d903669f1c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723436 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723471 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.723832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.723640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.724925 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.724901 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.727686 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.727638 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.728672 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.728645 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.729163 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.729120 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee25149-b18f-43d8-ab4e-e9d903669f1c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.730273 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.730236 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.731186 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.731164 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.732324 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.732283 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvck\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-kube-api-access-mnvck\") pod \"istiod-openshift-gateway-7b7995d5cd-8gxms\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.769340 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.768966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" event={"ID":"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec","Type":"ContainerStarted","Data":"70791db9a5ce8863b337829dada62a61215f56e4a18bb05fbb8f0e7790e7d09d"} Jul 27 23:42:10.770997 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.770930 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" event={"ID":"e8b6038f-3e24-4d26-87cd-215ee044673c","Type":"ContainerStarted","Data":"642128045b91f1a1411fa4a10b9be398616cf9d9cc45d6b99a954c361ae1654b"} Jul 27 23:42:10.790415 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.790383 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:10.959263 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:10.959236 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms"] Jul 27 23:42:10.961847 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:42:10.961817 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee25149_b18f_43d8_ab4e_e9d903669f1c.slice/crio-81a5e614f8d8f21a52472dfc659e22d402682ace2bf43dc9f57d0e420cd4417e WatchSource:0}: Error finding container 81a5e614f8d8f21a52472dfc659e22d402682ace2bf43dc9f57d0e420cd4417e: Status 404 returned error can't find the container with id 81a5e614f8d8f21a52472dfc659e22d402682ace2bf43dc9f57d0e420cd4417e Jul 27 23:42:11.465854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:11.465818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:11.777371 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:11.777328 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" event={"ID":"9ee25149-b18f-43d8-ab4e-e9d903669f1c","Type":"ContainerStarted","Data":"81a5e614f8d8f21a52472dfc659e22d402682ace2bf43dc9f57d0e420cd4417e"} Jul 27 23:42:13.790787 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.790742 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" event={"ID":"2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec","Type":"ContainerStarted","Data":"8a29cf79c313402b2aa0234946a4779fb42f1078d89aa9ed94197420cee1aeaa"} Jul 27 23:42:13.791260 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.790985 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:13.793919 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.793883 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" event={"ID":"e8b6038f-3e24-4d26-87cd-215ee044673c","Type":"ContainerStarted","Data":"817d5d4eb8614fa6cf26b1902c85a724d47fe19a98a003f48522cc74a5aec07f"} Jul 27 23:42:13.794109 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.794093 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:13.796105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.796037 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" event={"ID":"c8bc1753-f028-4a6a-97d1-021a0aad7b74","Type":"ContainerStarted","Data":"cb4abfbfec01952b1611c3ec1b71a2fcb3f2706b2afb9d3af60e50fdf90e065f"} Jul 27 23:42:13.819208 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.819153 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" podStartSLOduration=0.747908623 podStartE2EDuration="3.81913495s" totalImagesPullingTime="3.071226327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:13.805027199 +0000 UTC m=+333.729236078" watchObservedRunningTime="2026-07-27 23:42:13.81913495 +0000 UTC m=+333.743343831" Jul 27 23:42:13.819780 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.819738 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" podStartSLOduration=0.998351923 podStartE2EDuration="4.819725583s" totalImagesPullingTime="3.82137366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:13.817679399 +0000 UTC m=+333.741888278" watchObservedRunningTime="2026-07-27 23:42:13.819725583 +0000 UTC m=+333.743934462" Jul 27 23:42:13.831874 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:13.831831 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f44c56477-pjfvw" podStartSLOduration=0.877171729 podStartE2EDuration="4.831814182s" totalImagesPullingTime="3.954642453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:13.83101907 +0000 UTC m=+333.755227949" watchObservedRunningTime="2026-07-27 23:42:13.831814182 +0000 UTC m=+333.756023060" Jul 27 23:42:14.978289 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:14.978252 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 23:42:14.978549 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:14.978320 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 23:42:15.480808 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:15.480766 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:15.580647 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:15.580613 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:15.806372 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:15.806335 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" event={"ID":"9ee25149-b18f-43d8-ab4e-e9d903669f1c","Type":"ContainerStarted","Data":"5b0ca8e7bb95bc1cf09b087b2a4028fd97e79a38be1924b8e73ca2dc3c06fbcb"} Jul 27 23:42:15.806599 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:15.806463 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:15.859524 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:15.859462 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" podStartSLOduration=1.845505636 podStartE2EDuration="5.8594488s" totalImagesPullingTime="4.013943164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:15.851116015 +0000 UTC m=+335.775324895" watchObservedRunningTime="2026-07-27 23:42:15.8594488 +0000 UTC m=+335.783657677" Jul 27 23:42:16.258291 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:16.258245 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:16.543034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:16.542935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:16.758954 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:16.758913 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:16.811784 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:16.811713 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:18.934608 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.934522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:18.985442 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.985413 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k"] Jul 27 23:42:18.988257 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.988241 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:18.990225 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.990201 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 23:42:18.990328 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.990238 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-5djrd\"" Jul 27 23:42:18.994063 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.994041 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt"] Jul 27 23:42:18.994226 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.994097 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0be1c2-198c-477c-9762-7a9ddd914902-cert\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:18.994464 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.994440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4bq\" (UniqueName: \"kubernetes.io/projected/ea0be1c2-198c-477c-9762-7a9ddd914902-kube-api-access-fv4bq\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:18.998577 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.998558 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:18.999811 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:18.999787 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k"] Jul 27 23:42:19.000458 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.000435 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 23:42:19.000758 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.000737 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h97cq\"" Jul 27 23:42:19.008373 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.008352 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt"] Jul 27 23:42:19.021255 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.021217 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:19.101146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.096061 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkghx\" (UniqueName: \"kubernetes.io/projected/99ce8fe3-a611-4b1d-8dc6-c69c35823f2b-kube-api-access-vkghx\") pod \"llmisvc-controller-manager-777c6bd7d5-ld8xt\" (UID: \"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.101146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.096119 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0be1c2-198c-477c-9762-7a9ddd914902-cert\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:19.101146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.096174 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ce8fe3-a611-4b1d-8dc6-c69c35823f2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ld8xt\" (UID: \"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.101146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.096208 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4bq\" (UniqueName: \"kubernetes.io/projected/ea0be1c2-198c-477c-9762-7a9ddd914902-kube-api-access-fv4bq\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:19.101146 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:42:19.096701 2555 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 27 23:42:19.101146 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:42:19.096791 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0be1c2-198c-477c-9762-7a9ddd914902-cert podName:ea0be1c2-198c-477c-9762-7a9ddd914902 nodeName:}" failed. No retries permitted until 2026-07-27 23:42:19.596764273 +0000 UTC m=+339.520973133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ea0be1c2-198c-477c-9762-7a9ddd914902-cert") pod "kserve-controller-manager-d6f4b65b8-4tz9k" (UID: "ea0be1c2-198c-477c-9762-7a9ddd914902") : secret "kserve-webhook-server-cert" not found Jul 27 23:42:19.104607 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.104585 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4bq\" (UniqueName: \"kubernetes.io/projected/ea0be1c2-198c-477c-9762-7a9ddd914902-kube-api-access-fv4bq\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:19.196817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.196737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkghx\" (UniqueName: \"kubernetes.io/projected/99ce8fe3-a611-4b1d-8dc6-c69c35823f2b-kube-api-access-vkghx\") pod \"llmisvc-controller-manager-777c6bd7d5-ld8xt\" (UID: \"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.196817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.196795 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ce8fe3-a611-4b1d-8dc6-c69c35823f2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ld8xt\" (UID: \"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.199153 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.199131 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ce8fe3-a611-4b1d-8dc6-c69c35823f2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ld8xt\" (UID: \"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.209534 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.209489 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkghx\" (UniqueName: \"kubernetes.io/projected/99ce8fe3-a611-4b1d-8dc6-c69c35823f2b-kube-api-access-vkghx\") pod \"llmisvc-controller-manager-777c6bd7d5-ld8xt\" (UID: \"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.267551 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.267522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:19.310459 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.310429 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:19.370668 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.370450 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:19.448835 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.448759 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt"] Jul 27 23:42:19.451699 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:42:19.451671 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod99ce8fe3_a611_4b1d_8dc6_c69c35823f2b.slice/crio-235b898aca6c191ce2dc37a902d0404e52df4423e7d1ea6604f114a3249a5f92 WatchSource:0}: Error finding container 235b898aca6c191ce2dc37a902d0404e52df4423e7d1ea6604f114a3249a5f92: Status 404 returned error can't find the container with id 235b898aca6c191ce2dc37a902d0404e52df4423e7d1ea6604f114a3249a5f92 Jul 27 23:42:19.498728 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.498699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:19.599928 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.599896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0be1c2-198c-477c-9762-7a9ddd914902-cert\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:19.602259 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.602241 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea0be1c2-198c-477c-9762-7a9ddd914902-cert\") pod \"kserve-controller-manager-d6f4b65b8-4tz9k\" (UID: \"ea0be1c2-198c-477c-9762-7a9ddd914902\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:19.822515 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.822466 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" event={"ID":"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b","Type":"ContainerStarted","Data":"235b898aca6c191ce2dc37a902d0404e52df4423e7d1ea6604f114a3249a5f92"} Jul 27 23:42:19.900861 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:19.900824 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:20.029924 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:20.029892 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k"] Jul 27 23:42:20.069637 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:42:20.069606 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0be1c2_198c_477c_9762_7a9ddd914902.slice/crio-1197b67f511f9119ed46571331f2bab5bd69222410fbf81312d19a885c480800 WatchSource:0}: Error finding container 1197b67f511f9119ed46571331f2bab5bd69222410fbf81312d19a885c480800: Status 404 returned error can't find the container with id 1197b67f511f9119ed46571331f2bab5bd69222410fbf81312d19a885c480800 Jul 27 23:42:20.830094 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:20.830057 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" event={"ID":"ea0be1c2-198c-477c-9762-7a9ddd914902","Type":"ContainerStarted","Data":"1197b67f511f9119ed46571331f2bab5bd69222410fbf81312d19a885c480800"} Jul 27 23:42:21.701388 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:21.701352 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms"] Jul 27 23:42:21.701830 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:21.701564 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" podUID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" containerName="discovery" containerID="cri-o://5b0ca8e7bb95bc1cf09b087b2a4028fd97e79a38be1924b8e73ca2dc3c06fbcb" gracePeriod=30 Jul 27 23:42:21.837378 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:21.837348 2555 generic.go:356] "Generic (PLEG): container finished" podID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" containerID="5b0ca8e7bb95bc1cf09b087b2a4028fd97e79a38be1924b8e73ca2dc3c06fbcb" exitCode=0 Jul 27 23:42:21.837567 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:21.837400 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" event={"ID":"9ee25149-b18f-43d8-ab4e-e9d903669f1c","Type":"ContainerDied","Data":"5b0ca8e7bb95bc1cf09b087b2a4028fd97e79a38be1924b8e73ca2dc3c06fbcb"} Jul 27 23:42:22.855325 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.855302 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:22.928316 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928296 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-cacerts\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928416 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928336 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-dns-cert\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928416 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928402 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-token\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928531 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928417 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-kubeconfig\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928531 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928442 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-ca-configmap\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928531 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928481 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnvck\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-kube-api-access-mnvck\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928531 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928519 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee25149-b18f-43d8-ab4e-e9d903669f1c-local-certs\") pod \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\" (UID: \"9ee25149-b18f-43d8-ab4e-e9d903669f1c\") " Jul 27 23:42:22.928872 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.928832 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:42:22.931127 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.931066 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:42:22.931127 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.931084 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee25149-b18f-43d8-ab4e-e9d903669f1c-local-certs" (OuterVolumeSpecName: "local-certs") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:42:22.931127 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.931089 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:42:22.931127 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.931090 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:42:22.931127 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.931107 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-kube-api-access-mnvck" (OuterVolumeSpecName: "kube-api-access-mnvck") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "kube-api-access-mnvck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:42:22.931355 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:22.931179 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-cacerts" (OuterVolumeSpecName: "cacerts") pod "9ee25149-b18f-43d8-ab4e-e9d903669f1c" (UID: "9ee25149-b18f-43d8-ab4e-e9d903669f1c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:42:23.030029 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.029987 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-token\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.030029 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.030023 2555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-kubeconfig\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.030029 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.030032 2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-ca-configmap\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.030191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.030042 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnvck\" (UniqueName: \"kubernetes.io/projected/9ee25149-b18f-43d8-ab4e-e9d903669f1c-kube-api-access-mnvck\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.030191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.030051 2555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee25149-b18f-43d8-ab4e-e9d903669f1c-local-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.030191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.030059 2555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-cacerts\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.030191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.030066 2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ee25149-b18f-43d8-ab4e-e9d903669f1c-istio-csr-dns-cert\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:42:23.846934 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.846900 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" Jul 27 23:42:23.847104 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.846904 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" event={"ID":"9ee25149-b18f-43d8-ab4e-e9d903669f1c","Type":"ContainerDied","Data":"81a5e614f8d8f21a52472dfc659e22d402682ace2bf43dc9f57d0e420cd4417e"} Jul 27 23:42:23.847104 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.847035 2555 scope.go:117] "RemoveContainer" containerID="5b0ca8e7bb95bc1cf09b087b2a4028fd97e79a38be1924b8e73ca2dc3c06fbcb" Jul 27 23:42:23.848740 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.848627 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" event={"ID":"99ce8fe3-a611-4b1d-8dc6-c69c35823f2b","Type":"ContainerStarted","Data":"f3396e3abef4a034aa7e128791bc7e941a4773cffcd60da41889c6ddd679915d"} Jul 27 23:42:23.848740 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.848680 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:23.850908 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.850806 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" event={"ID":"ea0be1c2-198c-477c-9762-7a9ddd914902","Type":"ContainerStarted","Data":"60515f2c35c4a58debcff7a2afe2cffd06689237ffa59a0243aba297b2d6e35c"} Jul 27 23:42:23.855983 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.851218 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:42:23.872255 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.872206 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" podStartSLOduration=2.456029614 podStartE2EDuration="5.872194856s" totalImagesPullingTime="3.416165242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:23.871446454 +0000 UTC m=+343.795655333" watchObservedRunningTime="2026-07-27 23:42:23.872194856 +0000 UTC m=+343.796403733" Jul 27 23:42:23.893697 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.893656 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" podStartSLOduration=3.098732071 podStartE2EDuration="5.893642133s" totalImagesPullingTime="2.794910062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:23.892404942 +0000 UTC m=+343.816613821" watchObservedRunningTime="2026-07-27 23:42:23.893642133 +0000 UTC m=+343.817851010" Jul 27 23:42:23.910230 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.910207 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms"] Jul 27 23:42:23.914483 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:23.914463 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms"] Jul 27 23:42:24.564508 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:24.564459 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" path="/var/lib/kubelet/pods/9ee25149-b18f-43d8-ab4e-e9d903669f1c/volumes" Jul 27 23:42:24.803378 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:24.803348 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pnshl" Jul 27 23:42:24.805488 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:24.805466 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6dfjv" Jul 27 23:42:27.810663 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:27.810623 2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-8gxms container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.20:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 23:42:27.811049 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:27.810686 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8gxms" podUID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.20:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 23:42:50.049461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:50.049421 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:50.106424 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:50.106379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:54.857851 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:54.857824 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ld8xt" Jul 27 23:42:54.860858 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:42:54.860834 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4tz9k" Jul 27 23:43:08.930379 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:08.930337 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:09.033177 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.033129 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:09.033471 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.033450 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hr5wv"] Jul 27 23:43:09.033753 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.033736 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" containerName="discovery" Jul 27 23:43:09.033753 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.033749 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" containerName="discovery" Jul 27 23:43:09.033835 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.033819 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee25149-b18f-43d8-ab4e-e9d903669f1c" containerName="discovery" Jul 27 23:43:09.035746 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.035725 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:09.038340 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.038314 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 23:43:09.038454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.038349 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 23:43:09.039013 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.038992 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-tpsvv\"" Jul 27 23:43:09.047058 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.047037 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hr5wv"] Jul 27 23:43:09.118912 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.118881 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:09.191167 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.191092 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwzs\" (UniqueName: \"kubernetes.io/projected/adc73c65-913d-46df-976a-cd78f5d066a3-kube-api-access-mlwzs\") pod \"kuadrant-operator-catalog-hr5wv\" (UID: \"adc73c65-913d-46df-976a-cd78f5d066a3\") " pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:09.291969 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.291937 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwzs\" (UniqueName: \"kubernetes.io/projected/adc73c65-913d-46df-976a-cd78f5d066a3-kube-api-access-mlwzs\") pod \"kuadrant-operator-catalog-hr5wv\" (UID: \"adc73c65-913d-46df-976a-cd78f5d066a3\") " pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:09.299167 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.299145 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwzs\" (UniqueName: \"kubernetes.io/projected/adc73c65-913d-46df-976a-cd78f5d066a3-kube-api-access-mlwzs\") pod \"kuadrant-operator-catalog-hr5wv\" (UID: \"adc73c65-913d-46df-976a-cd78f5d066a3\") " pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:09.345189 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.345153 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:09.388291 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.388257 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hr5wv"] Jul 27 23:43:09.490079 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.489940 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hr5wv"] Jul 27 23:43:09.492869 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:43:09.492844 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc73c65_913d_46df_976a_cd78f5d066a3.slice/crio-b33469d2832157b93c7e989bcb47025c73985c74fe3664fbc33e940ddee5f106 WatchSource:0}: Error finding container b33469d2832157b93c7e989bcb47025c73985c74fe3664fbc33e940ddee5f106: Status 404 returned error can't find the container with id b33469d2832157b93c7e989bcb47025c73985c74fe3664fbc33e940ddee5f106 Jul 27 23:43:09.597522 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.597479 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v7dmc"] Jul 27 23:43:09.600646 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.600630 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:09.607610 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.607462 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v7dmc"] Jul 27 23:43:09.695375 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.695340 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fb2s\" (UniqueName: \"kubernetes.io/projected/e62024de-08cd-4702-b8e4-015f95b0a854-kube-api-access-8fb2s\") pod \"kuadrant-operator-catalog-v7dmc\" (UID: \"e62024de-08cd-4702-b8e4-015f95b0a854\") " pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:09.796188 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.796151 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fb2s\" (UniqueName: \"kubernetes.io/projected/e62024de-08cd-4702-b8e4-015f95b0a854-kube-api-access-8fb2s\") pod \"kuadrant-operator-catalog-v7dmc\" (UID: \"e62024de-08cd-4702-b8e4-015f95b0a854\") " pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:09.803294 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.803269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fb2s\" (UniqueName: \"kubernetes.io/projected/e62024de-08cd-4702-b8e4-015f95b0a854-kube-api-access-8fb2s\") pod \"kuadrant-operator-catalog-v7dmc\" (UID: \"e62024de-08cd-4702-b8e4-015f95b0a854\") " pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:09.823908 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.823876 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:09.911039 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:09.911007 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:10.019944 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:10.019891 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:10.032161 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:10.032121 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" event={"ID":"adc73c65-913d-46df-976a-cd78f5d066a3","Type":"ContainerStarted","Data":"b33469d2832157b93c7e989bcb47025c73985c74fe3664fbc33e940ddee5f106"} Jul 27 23:43:10.032380 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:10.032326 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v7dmc"] Jul 27 23:43:10.035067 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:43:10.035041 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62024de_08cd_4702_b8e4_015f95b0a854.slice/crio-220c42116e2b50954c41787ab775387fad1c3fc60a3f037fb03808867ce7cb56 WatchSource:0}: Error finding container 220c42116e2b50954c41787ab775387fad1c3fc60a3f037fb03808867ce7cb56: Status 404 returned error can't find the container with id 220c42116e2b50954c41787ab775387fad1c3fc60a3f037fb03808867ce7cb56 Jul 27 23:43:11.038797 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:11.038733 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" event={"ID":"e62024de-08cd-4702-b8e4-015f95b0a854","Type":"ContainerStarted","Data":"220c42116e2b50954c41787ab775387fad1c3fc60a3f037fb03808867ce7cb56"} Jul 27 23:43:12.043949 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.043907 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" event={"ID":"adc73c65-913d-46df-976a-cd78f5d066a3","Type":"ContainerStarted","Data":"e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b"} Jul 27 23:43:12.044406 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.043975 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" podUID="adc73c65-913d-46df-976a-cd78f5d066a3" containerName="registry-server" containerID="cri-o://e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b" gracePeriod=2 Jul 27 23:43:12.045423 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.045395 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" event={"ID":"e62024de-08cd-4702-b8e4-015f95b0a854","Type":"ContainerStarted","Data":"5926ba36a67b8df9b0a10c1e3e648fe6f69b5eeb41fc63cd24a441a55335b8af"} Jul 27 23:43:12.058639 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.058598 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" podStartSLOduration=1.061104385 podStartE2EDuration="3.058583238s" totalImagesPullingTime="1.997478853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:43:12.057377344 +0000 UTC m=+391.981586222" watchObservedRunningTime="2026-07-27 23:43:12.058583238 +0000 UTC m=+391.982792115" Jul 27 23:43:12.071218 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.071178 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" podStartSLOduration=1.604741925 podStartE2EDuration="3.071166946s" totalImagesPullingTime="1.466425021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:43:12.069837622 +0000 UTC m=+391.994046501" watchObservedRunningTime="2026-07-27 23:43:12.071166946 +0000 UTC m=+391.995375823" Jul 27 23:43:12.283864 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.283842 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:12.421887 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.421793 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlwzs\" (UniqueName: \"kubernetes.io/projected/adc73c65-913d-46df-976a-cd78f5d066a3-kube-api-access-mlwzs\") pod \"adc73c65-913d-46df-976a-cd78f5d066a3\" (UID: \"adc73c65-913d-46df-976a-cd78f5d066a3\") " Jul 27 23:43:12.423846 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.423814 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc73c65-913d-46df-976a-cd78f5d066a3-kube-api-access-mlwzs" (OuterVolumeSpecName: "kube-api-access-mlwzs") pod "adc73c65-913d-46df-976a-cd78f5d066a3" (UID: "adc73c65-913d-46df-976a-cd78f5d066a3"). InnerVolumeSpecName "kube-api-access-mlwzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:43:12.522328 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:12.522298 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwzs\" (UniqueName: \"kubernetes.io/projected/adc73c65-913d-46df-976a-cd78f5d066a3-kube-api-access-mlwzs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:43:13.050356 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.050319 2555 generic.go:356] "Generic (PLEG): container finished" podID="adc73c65-913d-46df-976a-cd78f5d066a3" containerID="e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b" exitCode=0 Jul 27 23:43:13.050812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.050361 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" Jul 27 23:43:13.050812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.050405 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" event={"ID":"adc73c65-913d-46df-976a-cd78f5d066a3","Type":"ContainerDied","Data":"e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b"} Jul 27 23:43:13.050812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.050459 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hr5wv" event={"ID":"adc73c65-913d-46df-976a-cd78f5d066a3","Type":"ContainerDied","Data":"b33469d2832157b93c7e989bcb47025c73985c74fe3664fbc33e940ddee5f106"} Jul 27 23:43:13.050812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.050480 2555 scope.go:117] "RemoveContainer" containerID="e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b" Jul 27 23:43:13.058948 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.058931 2555 scope.go:117] "RemoveContainer" containerID="e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b" Jul 27 23:43:13.059218 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:43:13.059196 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b\": container with ID starting with e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b not found: ID does not exist" containerID="e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b" Jul 27 23:43:13.059262 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.059227 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b"} err="failed to get container status \"e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b\": rpc error: code = NotFound desc = could not find container \"e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b\": container with ID starting with e526c77d0d4f157fcee52c4c8732fb9f55da2b09b2352b7f6cb930c8c45cc33b not found: ID does not exist" Jul 27 23:43:13.064007 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.063984 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hr5wv"] Jul 27 23:43:13.067279 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:13.067249 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hr5wv"] Jul 27 23:43:14.564197 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:14.564163 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc73c65-913d-46df-976a-cd78f5d066a3" path="/var/lib/kubelet/pods/adc73c65-913d-46df-976a-cd78f5d066a3/volumes" Jul 27 23:43:19.912177 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:19.912142 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:19.912600 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:19.912202 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:19.933269 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:19.933242 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:20.098461 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:20.098429 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-v7dmc" Jul 27 23:43:28.647255 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:28.647206 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:28.748041 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:28.747995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:30.606386 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:30.606343 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:30.671812 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:30.671751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:37.735757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:37.735719 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:38.332870 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:38.332819 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:41.809637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.809600 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dx68"] Jul 27 23:43:41.810027 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.809907 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc73c65-913d-46df-976a-cd78f5d066a3" containerName="registry-server" Jul 27 23:43:41.810027 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.809919 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc73c65-913d-46df-976a-cd78f5d066a3" containerName="registry-server" Jul 27 23:43:41.810027 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.809986 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc73c65-913d-46df-976a-cd78f5d066a3" containerName="registry-server" Jul 27 23:43:41.812713 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.812693 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:41.814922 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.814902 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-chzn5\"" Jul 27 23:43:41.821479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.821454 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dx68"] Jul 27 23:43:41.842638 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.842610 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rqk\" (UniqueName: \"kubernetes.io/projected/099eb9bd-acdf-4b70-ac5f-9d6010479968-kube-api-access-m9rqk\") pod \"authorino-operator-75cc8d5cb7-7dx68\" (UID: \"099eb9bd-acdf-4b70-ac5f-9d6010479968\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:41.943202 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.943163 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rqk\" (UniqueName: \"kubernetes.io/projected/099eb9bd-acdf-4b70-ac5f-9d6010479968-kube-api-access-m9rqk\") pod \"authorino-operator-75cc8d5cb7-7dx68\" (UID: \"099eb9bd-acdf-4b70-ac5f-9d6010479968\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:41.955687 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:41.955658 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rqk\" (UniqueName: \"kubernetes.io/projected/099eb9bd-acdf-4b70-ac5f-9d6010479968-kube-api-access-m9rqk\") pod \"authorino-operator-75cc8d5cb7-7dx68\" (UID: \"099eb9bd-acdf-4b70-ac5f-9d6010479968\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:42.123055 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:42.122969 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:42.254304 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:42.254282 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dx68"] Jul 27 23:43:42.256910 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:43:42.256877 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099eb9bd_acdf_4b70_ac5f_9d6010479968.slice/crio-8b0d07f8c72aaa45e2700b577e1185f21f441ba9d5710e294522d21432c81345 WatchSource:0}: Error finding container 8b0d07f8c72aaa45e2700b577e1185f21f441ba9d5710e294522d21432c81345: Status 404 returned error can't find the container with id 8b0d07f8c72aaa45e2700b577e1185f21f441ba9d5710e294522d21432c81345 Jul 27 23:43:42.733767 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:42.733735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:43.172955 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:43.172915 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" event={"ID":"099eb9bd-acdf-4b70-ac5f-9d6010479968","Type":"ContainerStarted","Data":"8b0d07f8c72aaa45e2700b577e1185f21f441ba9d5710e294522d21432c81345"} Jul 27 23:43:44.968823 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:44.968727 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:45.084200 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:45.084164 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:45.182476 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:45.182448 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" event={"ID":"099eb9bd-acdf-4b70-ac5f-9d6010479968","Type":"ContainerStarted","Data":"5a114ea42827bb1214ecee52cbcd2b85c6a2c67e2a248c46e4a5175a1c5e6b4f"} Jul 27 23:43:45.182637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:45.182597 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:45.199934 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:45.199881 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" podStartSLOduration=1.9880273800000001 podStartE2EDuration="4.19986657s" totalImagesPullingTime="2.21183919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:43:45.197917554 +0000 UTC m=+425.122126433" watchObservedRunningTime="2026-07-27 23:43:45.19986657 +0000 UTC m=+425.124075448" Jul 27 23:43:46.869882 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:46.869851 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8"] Jul 27 23:43:46.876328 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:46.876307 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:46.878358 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:46.878340 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q94k8\"" Jul 27 23:43:46.883570 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:46.883548 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8"] Jul 27 23:43:46.980472 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:46.980443 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qz4d\" (UniqueName: \"kubernetes.io/projected/c9a3ce06-733d-4959-b974-bfdf1bee753d-kube-api-access-2qz4d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:46.980605 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:46.980514 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a3ce06-733d-4959-b974-bfdf1bee753d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:47.081826 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.081784 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qz4d\" (UniqueName: \"kubernetes.io/projected/c9a3ce06-733d-4959-b974-bfdf1bee753d-kube-api-access-2qz4d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:47.081954 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.081861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a3ce06-733d-4959-b974-bfdf1bee753d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:47.082196 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.082180 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a3ce06-733d-4959-b974-bfdf1bee753d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:47.090613 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.090586 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qz4d\" (UniqueName: \"kubernetes.io/projected/c9a3ce06-733d-4959-b974-bfdf1bee753d-kube-api-access-2qz4d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:47.186780 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.186716 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:47.317257 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.317229 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8"] Jul 27 23:43:47.319806 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:43:47.319778 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a3ce06_733d_4959_b974_bfdf1bee753d.slice/crio-3551c917556619c43cef829729eebeb54828b2b1f4ab6cc148fe8c6586a01e32 WatchSource:0}: Error finding container 3551c917556619c43cef829729eebeb54828b2b1f4ab6cc148fe8c6586a01e32: Status 404 returned error can't find the container with id 3551c917556619c43cef829729eebeb54828b2b1f4ab6cc148fe8c6586a01e32 Jul 27 23:43:47.727052 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.727023 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:47.945269 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:47.945228 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:48.195796 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:48.195755 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" event={"ID":"c9a3ce06-733d-4959-b974-bfdf1bee753d","Type":"ContainerStarted","Data":"3551c917556619c43cef829729eebeb54828b2b1f4ab6cc148fe8c6586a01e32"} Jul 27 23:43:49.775631 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:49.775592 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24"] Jul 27 23:43:49.778798 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:49.778777 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:43:49.780980 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:49.780945 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 27 23:43:49.781082 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:49.781050 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v95lv\"" Jul 27 23:43:49.786361 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:49.786339 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24"] Jul 27 23:43:49.902101 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:49.902064 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8vs\" (UniqueName: \"kubernetes.io/projected/e071e05a-5504-4633-ace8-135a5010c156-kube-api-access-tz8vs\") pod \"dns-operator-controller-manager-864dd78c4c-zgm24\" (UID: \"e071e05a-5504-4633-ace8-135a5010c156\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:43:50.002665 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:50.002631 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8vs\" (UniqueName: \"kubernetes.io/projected/e071e05a-5504-4633-ace8-135a5010c156-kube-api-access-tz8vs\") pod \"dns-operator-controller-manager-864dd78c4c-zgm24\" (UID: \"e071e05a-5504-4633-ace8-135a5010c156\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:43:50.010743 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:50.010720 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8vs\" (UniqueName: \"kubernetes.io/projected/e071e05a-5504-4633-ace8-135a5010c156-kube-api-access-tz8vs\") pod \"dns-operator-controller-manager-864dd78c4c-zgm24\" (UID: \"e071e05a-5504-4633-ace8-135a5010c156\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:43:50.089208 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:50.089099 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:43:50.282028 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:50.281992 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24"] Jul 27 23:43:50.283266 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:43:50.283235 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode071e05a_5504_4633_ace8_135a5010c156.slice/crio-42d5d5e4813c27b59236bbadf15a2293c55dadf8529ad8282ef4668768e30f6f WatchSource:0}: Error finding container 42d5d5e4813c27b59236bbadf15a2293c55dadf8529ad8282ef4668768e30f6f: Status 404 returned error can't find the container with id 42d5d5e4813c27b59236bbadf15a2293c55dadf8529ad8282ef4668768e30f6f Jul 27 23:43:50.372155 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:50.372077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:50.991083 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:50.991038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:51.214381 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:51.214351 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" event={"ID":"e071e05a-5504-4633-ace8-135a5010c156","Type":"ContainerStarted","Data":"42d5d5e4813c27b59236bbadf15a2293c55dadf8529ad8282ef4668768e30f6f"} Jul 27 23:43:52.196407 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.196378 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z"] Jul 27 23:43:52.199898 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.199873 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:43:52.202146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.202122 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qgkl9\"" Jul 27 23:43:52.211367 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.211307 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z"] Jul 27 23:43:52.322963 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.322924 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5p2\" (UniqueName: \"kubernetes.io/projected/0316117e-1c6e-4c23-a95c-aae81b00782b-kube-api-access-cw5p2\") pod \"limitador-operator-controller-manager-859d7b78bd-8f57z\" (UID: \"0316117e-1c6e-4c23-a95c-aae81b00782b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:43:52.423815 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.423777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5p2\" (UniqueName: \"kubernetes.io/projected/0316117e-1c6e-4c23-a95c-aae81b00782b-kube-api-access-cw5p2\") pod \"limitador-operator-controller-manager-859d7b78bd-8f57z\" (UID: \"0316117e-1c6e-4c23-a95c-aae81b00782b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:43:52.432557 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.432525 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5p2\" (UniqueName: \"kubernetes.io/projected/0316117e-1c6e-4c23-a95c-aae81b00782b-kube-api-access-cw5p2\") pod \"limitador-operator-controller-manager-859d7b78bd-8f57z\" (UID: \"0316117e-1c6e-4c23-a95c-aae81b00782b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:43:52.514885 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.514845 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:43:52.818995 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:52.818869 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z"] Jul 27 23:43:53.026274 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:43:53.026233 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0316117e_1c6e_4c23_a95c_aae81b00782b.slice/crio-7d9f6007b701d11aa10dc3156c37db387d43ca74c6e1b2dd17f133d011bf9a7e WatchSource:0}: Error finding container 7d9f6007b701d11aa10dc3156c37db387d43ca74c6e1b2dd17f133d011bf9a7e: Status 404 returned error can't find the container with id 7d9f6007b701d11aa10dc3156c37db387d43ca74c6e1b2dd17f133d011bf9a7e Jul 27 23:43:53.224640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.224603 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" event={"ID":"e071e05a-5504-4633-ace8-135a5010c156","Type":"ContainerStarted","Data":"bf122afb35882b7f8981f1b53981f1bec7f310d60e4f73ef2a9f63083ca2474f"} Jul 27 23:43:53.225042 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.224700 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:43:53.225832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.225809 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" event={"ID":"0316117e-1c6e-4c23-a95c-aae81b00782b","Type":"ContainerStarted","Data":"7d9f6007b701d11aa10dc3156c37db387d43ca74c6e1b2dd17f133d011bf9a7e"} Jul 27 23:43:53.227237 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.227215 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" event={"ID":"c9a3ce06-733d-4959-b974-bfdf1bee753d","Type":"ContainerStarted","Data":"ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b"} Jul 27 23:43:53.227336 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.227273 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:43:53.248485 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.248442 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" podStartSLOduration=1.459006751 podStartE2EDuration="4.248422273s" totalImagesPullingTime="2.789415522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:43:53.247327789 +0000 UTC m=+433.171536667" watchObservedRunningTime="2026-07-27 23:43:53.248422273 +0000 UTC m=+433.172631151" Jul 27 23:43:53.267625 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.267583 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" podStartSLOduration=1.563899566 podStartE2EDuration="7.267570856s" totalImagesPullingTime="5.70367129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:43:53.265944542 +0000 UTC m=+433.190153421" watchObservedRunningTime="2026-07-27 23:43:53.267570856 +0000 UTC m=+433.191779734" Jul 27 23:43:53.384120 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:53.384033 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:54.145465 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:54.145422 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:54.246139 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:54.246091 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:54.535796 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:54.535761 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:54.591277 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:54.591242 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:55.238097 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:55.238066 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" event={"ID":"0316117e-1c6e-4c23-a95c-aae81b00782b","Type":"ContainerStarted","Data":"a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83"} Jul 27 23:43:55.238277 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:55.238165 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:43:55.949267 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:55.949235 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:56.189226 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:56.189190 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dx68" Jul 27 23:43:56.207868 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:56.207788 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" podStartSLOduration=2.526410395 podStartE2EDuration="4.207774918s" totalImagesPullingTime="1.681364523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:43:55.260043004 +0000 UTC m=+435.184251883" watchObservedRunningTime="2026-07-27 23:43:56.207774918 +0000 UTC m=+436.131983798" Jul 27 23:43:56.406372 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:56.406339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:57.945687 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:57.945652 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:43:58.042152 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:43:58.042114 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:04.235034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:04.234956 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zgm24" Jul 27 23:44:04.235034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:04.235031 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:05.217224 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:05.217177 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:05.334971 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:05.334940 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:06.210842 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.210808 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8"] Jul 27 23:44:06.211018 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.210993 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" containerName="manager" containerID="cri-o://ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b" gracePeriod=2 Jul 27 23:44:06.222226 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.222162 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8"] Jul 27 23:44:06.228807 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.228777 2555 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c9a3ce06-733d-4959-b974-bfdf1bee753d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\": pods \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:06.234046 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.234024 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn"] Jul 27 23:44:06.234541 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.234526 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" containerName="manager" Jul 27 23:44:06.234587 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.234545 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" containerName="manager" Jul 27 23:44:06.234644 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.234633 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" containerName="manager" Jul 27 23:44:06.237652 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.237635 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z"] Jul 27 23:44:06.237764 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.237746 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.237817 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.237795 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" containerName="manager" containerID="cri-o://a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83" gracePeriod=2 Jul 27 23:44:06.239748 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.239728 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:06.245040 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.245010 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z"] Jul 27 23:44:06.251227 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.251200 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn"] Jul 27 23:44:06.265122 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.265100 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4"] Jul 27 23:44:06.265404 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.265392 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" containerName="manager" Jul 27 23:44:06.265404 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.265405 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" containerName="manager" Jul 27 23:44:06.265526 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.265452 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" containerName="manager" Jul 27 23:44:06.268324 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.268302 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:06.288236 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.287765 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4"] Jul 27 23:44:06.288236 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.288121 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8f57z\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:06.290942 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.290919 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:06.292683 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.292654 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:06.310820 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.310789 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8f57z\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:06.337093 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.337068 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzm95\" (UniqueName: \"kubernetes.io/projected/0a7ffbf8-2622-4788-a433-e3b1caa9559d-kube-api-access-xzm95\") pod \"limitador-operator-controller-manager-859d7b78bd-blxc4\" (UID: \"0a7ffbf8-2622-4788-a433-e3b1caa9559d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:06.337411 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.337114 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/add66887-def6-4236-88f6-7f47836428b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4mkwn\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.337411 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.337173 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978sh\" (UniqueName: \"kubernetes.io/projected/add66887-def6-4236-88f6-7f47836428b5-kube-api-access-978sh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4mkwn\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.438054 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.438016 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzm95\" (UniqueName: \"kubernetes.io/projected/0a7ffbf8-2622-4788-a433-e3b1caa9559d-kube-api-access-xzm95\") pod \"limitador-operator-controller-manager-859d7b78bd-blxc4\" (UID: \"0a7ffbf8-2622-4788-a433-e3b1caa9559d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:06.438193 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.438073 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/add66887-def6-4236-88f6-7f47836428b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4mkwn\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.438193 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.438113 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978sh\" (UniqueName: \"kubernetes.io/projected/add66887-def6-4236-88f6-7f47836428b5-kube-api-access-978sh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4mkwn\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.438474 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.438452 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/add66887-def6-4236-88f6-7f47836428b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4mkwn\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.458061 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.458031 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzm95\" (UniqueName: \"kubernetes.io/projected/0a7ffbf8-2622-4788-a433-e3b1caa9559d-kube-api-access-xzm95\") pod \"limitador-operator-controller-manager-859d7b78bd-blxc4\" (UID: \"0a7ffbf8-2622-4788-a433-e3b1caa9559d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:06.458641 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.458622 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-978sh\" (UniqueName: \"kubernetes.io/projected/add66887-def6-4236-88f6-7f47836428b5-kube-api-access-978sh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4mkwn\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.484586 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.484565 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:06.486553 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.486537 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:06.491603 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.491576 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8f57z\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:06.493895 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.493874 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:06.495947 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.495928 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8f57z\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:06.498084 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:06.498063 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvlh8\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:06.539380 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.539348 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw5p2\" (UniqueName: \"kubernetes.io/projected/0316117e-1c6e-4c23-a95c-aae81b00782b-kube-api-access-cw5p2\") pod \"0316117e-1c6e-4c23-a95c-aae81b00782b\" (UID: \"0316117e-1c6e-4c23-a95c-aae81b00782b\") " Jul 27 23:44:06.539525 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.539385 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a3ce06-733d-4959-b974-bfdf1bee753d-extensions-socket-volume\") pod \"c9a3ce06-733d-4959-b974-bfdf1bee753d\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " Jul 27 23:44:06.539525 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.539452 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qz4d\" (UniqueName: \"kubernetes.io/projected/c9a3ce06-733d-4959-b974-bfdf1bee753d-kube-api-access-2qz4d\") pod \"c9a3ce06-733d-4959-b974-bfdf1bee753d\" (UID: \"c9a3ce06-733d-4959-b974-bfdf1bee753d\") " Jul 27 23:44:06.539830 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.539808 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a3ce06-733d-4959-b974-bfdf1bee753d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c9a3ce06-733d-4959-b974-bfdf1bee753d" (UID: "c9a3ce06-733d-4959-b974-bfdf1bee753d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:44:06.541488 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.541469 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316117e-1c6e-4c23-a95c-aae81b00782b-kube-api-access-cw5p2" (OuterVolumeSpecName: "kube-api-access-cw5p2") pod "0316117e-1c6e-4c23-a95c-aae81b00782b" (UID: "0316117e-1c6e-4c23-a95c-aae81b00782b"). InnerVolumeSpecName "kube-api-access-cw5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:44:06.541567 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.541479 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a3ce06-733d-4959-b974-bfdf1bee753d-kube-api-access-2qz4d" (OuterVolumeSpecName: "kube-api-access-2qz4d") pod "c9a3ce06-733d-4959-b974-bfdf1bee753d" (UID: "c9a3ce06-733d-4959-b974-bfdf1bee753d"). InnerVolumeSpecName "kube-api-access-2qz4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:44:06.565271 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.565245 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0316117e-1c6e-4c23-a95c-aae81b00782b" path="/var/lib/kubelet/pods/0316117e-1c6e-4c23-a95c-aae81b00782b/volumes" Jul 27 23:44:06.565602 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.565588 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a3ce06-733d-4959-b974-bfdf1bee753d" path="/var/lib/kubelet/pods/c9a3ce06-733d-4959-b974-bfdf1bee753d/volumes" Jul 27 23:44:06.640511 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.640464 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw5p2\" (UniqueName: \"kubernetes.io/projected/0316117e-1c6e-4c23-a95c-aae81b00782b-kube-api-access-cw5p2\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:44:06.640511 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.640512 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a3ce06-733d-4959-b974-bfdf1bee753d-extensions-socket-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:44:06.640671 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.640528 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qz4d\" (UniqueName: \"kubernetes.io/projected/c9a3ce06-733d-4959-b974-bfdf1bee753d-kube-api-access-2qz4d\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:44:06.655393 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.655371 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:06.662401 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.662375 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:06.820332 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.820307 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn"] Jul 27 23:44:06.822708 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:44:06.822675 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd66887_def6_4236_88f6_7f47836428b5.slice/crio-9feb30a1c7fb2b278345168586ae1784c33a8c5a48b69548d19b8e199606883a WatchSource:0}: Error finding container 9feb30a1c7fb2b278345168586ae1784c33a8c5a48b69548d19b8e199606883a: Status 404 returned error can't find the container with id 9feb30a1c7fb2b278345168586ae1784c33a8c5a48b69548d19b8e199606883a Jul 27 23:44:06.832877 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:06.832853 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4"] Jul 27 23:44:06.835653 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:44:06.835631 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7ffbf8_2622_4788_a433_e3b1caa9559d.slice/crio-ee7e06a12e8bc01cbe207ee6dfef91e402178272a3280c4a5ffa85fc5175edfb WatchSource:0}: Error finding container ee7e06a12e8bc01cbe207ee6dfef91e402178272a3280c4a5ffa85fc5175edfb: Status 404 returned error can't find the container with id ee7e06a12e8bc01cbe207ee6dfef91e402178272a3280c4a5ffa85fc5175edfb Jul 27 23:44:07.294551 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.294517 2555 generic.go:356] "Generic (PLEG): container finished" podID="0316117e-1c6e-4c23-a95c-aae81b00782b" containerID="a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83" exitCode=0 Jul 27 23:44:07.294733 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.294625 2555 scope.go:117] "RemoveContainer" containerID="a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83" Jul 27 23:44:07.294733 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.294626 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f57z" Jul 27 23:44:07.295987 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.295966 2555 generic.go:356] "Generic (PLEG): container finished" podID="c9a3ce06-733d-4959-b974-bfdf1bee753d" containerID="ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b" exitCode=0 Jul 27 23:44:07.296065 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.296019 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvlh8" Jul 27 23:44:07.297886 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.297860 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" event={"ID":"0a7ffbf8-2622-4788-a433-e3b1caa9559d","Type":"ContainerStarted","Data":"e2d6b06e7ed490d72ea27e5d022a82911758a3aeea7bdb11a6c81bf6543ee48f"} Jul 27 23:44:07.298004 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.297908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" event={"ID":"0a7ffbf8-2622-4788-a433-e3b1caa9559d","Type":"ContainerStarted","Data":"ee7e06a12e8bc01cbe207ee6dfef91e402178272a3280c4a5ffa85fc5175edfb"} Jul 27 23:44:07.298004 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.297964 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:07.299627 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.299595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" event={"ID":"add66887-def6-4236-88f6-7f47836428b5","Type":"ContainerStarted","Data":"300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775"} Jul 27 23:44:07.299737 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.299642 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" event={"ID":"add66887-def6-4236-88f6-7f47836428b5","Type":"ContainerStarted","Data":"9feb30a1c7fb2b278345168586ae1784c33a8c5a48b69548d19b8e199606883a"} Jul 27 23:44:07.299737 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.299670 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:07.303469 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.303453 2555 scope.go:117] "RemoveContainer" containerID="a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83" Jul 27 23:44:07.303724 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:07.303704 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83\": container with ID starting with a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83 not found: ID does not exist" containerID="a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83" Jul 27 23:44:07.303794 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.303730 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83"} err="failed to get container status \"a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83\": rpc error: code = NotFound desc = could not find container \"a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83\": container with ID starting with a1e80dc4098ba01a004eddc01e7e763e5f1cc9070289b9eac9307a5b82d9ee83 not found: ID does not exist" Jul 27 23:44:07.303794 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.303747 2555 scope.go:117] "RemoveContainer" containerID="ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b" Jul 27 23:44:07.311295 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.311280 2555 scope.go:117] "RemoveContainer" containerID="ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b" Jul 27 23:44:07.311588 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:07.311566 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b\": container with ID starting with ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b not found: ID does not exist" containerID="ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b" Jul 27 23:44:07.311684 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.311593 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b"} err="failed to get container status \"ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b\": rpc error: code = NotFound desc = could not find container \"ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b\": container with ID starting with ad2cbc6bebe9b795fe6747f95c643e4858cef2ef460b5d3cbaff99c31435a43b not found: ID does not exist" Jul 27 23:44:07.313644 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.313604 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" podStartSLOduration=1.3135910370000001 podStartE2EDuration="1.313591037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:44:07.312760396 +0000 UTC m=+447.236969277" watchObservedRunningTime="2026-07-27 23:44:07.313591037 +0000 UTC m=+447.237799980" Jul 27 23:44:07.331971 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:07.331930 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" podStartSLOduration=1.331921239 podStartE2EDuration="1.331921239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:44:07.330647701 +0000 UTC m=+447.254856581" watchObservedRunningTime="2026-07-27 23:44:07.331921239 +0000 UTC m=+447.256130117" Jul 27 23:44:09.355743 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:09.355708 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:09.456792 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:09.456748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:13.150280 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:13.150242 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:13.208303 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:13.208268 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:13.351345 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:13.351304 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:13.451788 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:13.451672 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:15.878409 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:15.878370 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:15.978645 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:15.978606 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:18.309276 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:18.309239 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:18.309757 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:18.309328 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-blxc4" Jul 27 23:44:22.354763 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:22.354726 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:22.456034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:22.455994 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:27.112689 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:27.112653 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:27.212479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:27.212449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:30.918244 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:30.918206 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:31.018705 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:31.018662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:31.516480 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:31.516440 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:31.616421 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:31.616375 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:35.320020 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.319984 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:35.376027 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.375997 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn"] Jul 27 23:44:35.376207 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.376183 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" podUID="add66887-def6-4236-88f6-7f47836428b5" containerName="manager" containerID="cri-o://300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775" gracePeriod=10 Jul 27 23:44:35.401579 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.401546 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:35.614044 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.614022 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:35.648244 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.648214 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl"] Jul 27 23:44:35.648724 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.648705 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add66887-def6-4236-88f6-7f47836428b5" containerName="manager" Jul 27 23:44:35.648775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.648727 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="add66887-def6-4236-88f6-7f47836428b5" containerName="manager" Jul 27 23:44:35.648825 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.648816 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="add66887-def6-4236-88f6-7f47836428b5" containerName="manager" Jul 27 23:44:35.650944 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.650921 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.663063 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.663042 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl"] Jul 27 23:44:35.677804 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.677780 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-978sh\" (UniqueName: \"kubernetes.io/projected/add66887-def6-4236-88f6-7f47836428b5-kube-api-access-978sh\") pod \"add66887-def6-4236-88f6-7f47836428b5\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " Jul 27 23:44:35.677902 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.677833 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/add66887-def6-4236-88f6-7f47836428b5-extensions-socket-volume\") pod \"add66887-def6-4236-88f6-7f47836428b5\" (UID: \"add66887-def6-4236-88f6-7f47836428b5\") " Jul 27 23:44:35.677953 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.677904 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6pf\" (UniqueName: \"kubernetes.io/projected/1463f956-55d1-4d41-8f37-17cfc2de87c8-kube-api-access-8c6pf\") pod \"kuadrant-operator-controller-manager-86878c769f-zj7kl\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.677990 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.677949 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1463f956-55d1-4d41-8f37-17cfc2de87c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zj7kl\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.678197 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.678170 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add66887-def6-4236-88f6-7f47836428b5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "add66887-def6-4236-88f6-7f47836428b5" (UID: "add66887-def6-4236-88f6-7f47836428b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:44:35.679941 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.679919 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add66887-def6-4236-88f6-7f47836428b5-kube-api-access-978sh" (OuterVolumeSpecName: "kube-api-access-978sh") pod "add66887-def6-4236-88f6-7f47836428b5" (UID: "add66887-def6-4236-88f6-7f47836428b5"). InnerVolumeSpecName "kube-api-access-978sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:44:35.778362 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.778326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1463f956-55d1-4d41-8f37-17cfc2de87c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zj7kl\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.778530 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.778376 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6pf\" (UniqueName: \"kubernetes.io/projected/1463f956-55d1-4d41-8f37-17cfc2de87c8-kube-api-access-8c6pf\") pod \"kuadrant-operator-controller-manager-86878c769f-zj7kl\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.778530 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.778436 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978sh\" (UniqueName: \"kubernetes.io/projected/add66887-def6-4236-88f6-7f47836428b5-kube-api-access-978sh\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:44:35.778530 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.778450 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/add66887-def6-4236-88f6-7f47836428b5-extensions-socket-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:44:35.778720 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.778700 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1463f956-55d1-4d41-8f37-17cfc2de87c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zj7kl\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.806035 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.806011 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6pf\" (UniqueName: \"kubernetes.io/projected/1463f956-55d1-4d41-8f37-17cfc2de87c8-kube-api-access-8c6pf\") pod \"kuadrant-operator-controller-manager-86878c769f-zj7kl\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:35.960262 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:35.960171 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:36.089210 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.089160 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl"] Jul 27 23:44:36.091856 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:44:36.091831 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1463f956_55d1_4d41_8f37_17cfc2de87c8.slice/crio-8ac6ed72fc5f152fad718ddaf0fcb509126bf32680010ce69be316dfbcaa3e4c WatchSource:0}: Error finding container 8ac6ed72fc5f152fad718ddaf0fcb509126bf32680010ce69be316dfbcaa3e4c: Status 404 returned error can't find the container with id 8ac6ed72fc5f152fad718ddaf0fcb509126bf32680010ce69be316dfbcaa3e4c Jul 27 23:44:36.426438 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.426408 2555 generic.go:356] "Generic (PLEG): container finished" podID="add66887-def6-4236-88f6-7f47836428b5" containerID="300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775" exitCode=0 Jul 27 23:44:36.427015 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.426461 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" Jul 27 23:44:36.427015 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.426477 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" event={"ID":"add66887-def6-4236-88f6-7f47836428b5","Type":"ContainerDied","Data":"300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775"} Jul 27 23:44:36.427015 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.426544 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn" event={"ID":"add66887-def6-4236-88f6-7f47836428b5","Type":"ContainerDied","Data":"9feb30a1c7fb2b278345168586ae1784c33a8c5a48b69548d19b8e199606883a"} Jul 27 23:44:36.427015 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.426569 2555 scope.go:117] "RemoveContainer" containerID="300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775" Jul 27 23:44:36.431134 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.428217 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" event={"ID":"1463f956-55d1-4d41-8f37-17cfc2de87c8","Type":"ContainerStarted","Data":"58dffd0cf74f8f6552b8e73f35482c19e637018af31afb5b23ae597e69c5fbee"} Jul 27 23:44:36.431134 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.428521 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" event={"ID":"1463f956-55d1-4d41-8f37-17cfc2de87c8","Type":"ContainerStarted","Data":"8ac6ed72fc5f152fad718ddaf0fcb509126bf32680010ce69be316dfbcaa3e4c"} Jul 27 23:44:36.431134 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.429203 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:36.439699 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.439580 2555 scope.go:117] "RemoveContainer" containerID="300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775" Jul 27 23:44:36.439913 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:44:36.439891 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775\": container with ID starting with 300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775 not found: ID does not exist" containerID="300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775" Jul 27 23:44:36.439975 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.439920 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775"} err="failed to get container status \"300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775\": rpc error: code = NotFound desc = could not find container \"300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775\": container with ID starting with 300e33db04be9c883d6535359914e64aa1cb8a73a42e71a2821103a6fa9fb775 not found: ID does not exist" Jul 27 23:44:36.449345 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.449307 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" podStartSLOduration=1.449295567 podStartE2EDuration="1.449295567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:44:36.448371882 +0000 UTC m=+476.372580759" watchObservedRunningTime="2026-07-27 23:44:36.449295567 +0000 UTC m=+476.373504445" Jul 27 23:44:36.463966 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.463945 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn"] Jul 27 23:44:36.477650 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.477623 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4mkwn"] Jul 27 23:44:36.564265 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:36.564232 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add66887-def6-4236-88f6-7f47836428b5" path="/var/lib/kubelet/pods/add66887-def6-4236-88f6-7f47836428b5/volumes" Jul 27 23:44:41.381107 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:41.381072 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:41.438463 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:41.438429 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:46.130198 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:46.130153 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:46.231047 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:46.231004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:48.226758 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:48.226724 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:48.440821 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:48.440789 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:44:51.713410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:51.713366 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:51.978261 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:51.978170 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:52.127105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:52.127072 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:52.226733 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:52.226701 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:55.412442 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.412396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:55.415458 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.415434 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:44:55.420146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.420128 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.422200 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.422181 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 23:44:55.422269 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.422210 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hwkjm\"" Jul 27 23:44:55.426041 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.426020 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:44:55.519902 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.519867 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:44:55.531611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.531575 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:55.534565 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.534543 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/15412a18-3323-4d49-b43a-6f219f7c977a-config-file\") pod \"limitador-limitador-64c94fd64f-49dwk\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.534685 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.534585 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm76b\" (UniqueName: \"kubernetes.io/projected/15412a18-3323-4d49-b43a-6f219f7c977a-kube-api-access-lm76b\") pod \"limitador-limitador-64c94fd64f-49dwk\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.586089 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.586052 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:55.635203 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.635174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/15412a18-3323-4d49-b43a-6f219f7c977a-config-file\") pod \"limitador-limitador-64c94fd64f-49dwk\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.635332 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.635215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm76b\" (UniqueName: \"kubernetes.io/projected/15412a18-3323-4d49-b43a-6f219f7c977a-kube-api-access-lm76b\") pod \"limitador-limitador-64c94fd64f-49dwk\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.635858 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.635838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/15412a18-3323-4d49-b43a-6f219f7c977a-config-file\") pod \"limitador-limitador-64c94fd64f-49dwk\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.642146 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.642124 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm76b\" (UniqueName: \"kubernetes.io/projected/15412a18-3323-4d49-b43a-6f219f7c977a-kube-api-access-lm76b\") pod \"limitador-limitador-64c94fd64f-49dwk\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.731986 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.731917 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:55.863096 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:55.862830 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:44:56.201060 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.201024 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-xv85c"] Jul 27 23:44:56.205483 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.205458 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:44:56.207540 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.207519 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b5mc4\"" Jul 27 23:44:56.212994 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.212969 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xv85c"] Jul 27 23:44:56.237611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.237586 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:56.239703 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.239682 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs25\" (UniqueName: \"kubernetes.io/projected/ecc528bb-fcae-4f94-aa61-ce86ce490cf2-kube-api-access-vcs25\") pod \"authorino-58cd955675-xv85c\" (UID: \"ecc528bb-fcae-4f94-aa61-ce86ce490cf2\") " pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:44:56.337138 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.337102 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:56.340169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.340141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs25\" (UniqueName: \"kubernetes.io/projected/ecc528bb-fcae-4f94-aa61-ce86ce490cf2-kube-api-access-vcs25\") pod \"authorino-58cd955675-xv85c\" (UID: \"ecc528bb-fcae-4f94-aa61-ce86ce490cf2\") " pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:44:56.348951 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.348932 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs25\" (UniqueName: \"kubernetes.io/projected/ecc528bb-fcae-4f94-aa61-ce86ce490cf2-kube-api-access-vcs25\") pod \"authorino-58cd955675-xv85c\" (UID: \"ecc528bb-fcae-4f94-aa61-ce86ce490cf2\") " pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:44:56.515384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.515354 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:44:56.516174 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.515830 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" event={"ID":"15412a18-3323-4d49-b43a-6f219f7c977a","Type":"ContainerStarted","Data":"085b6ecc16330ddfad14754f30d4e8d3c58b3018acde12197a938bcac5ef6ad6"} Jul 27 23:44:56.700486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:56.700414 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xv85c"] Jul 27 23:44:56.705983 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:44:56.705941 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecc528bb_fcae_4f94_aa61_ce86ce490cf2.slice/crio-04d53fddd58f22bf5db28bf8a60c022cf96181705ca36ba1c9f47c47e8100a25 WatchSource:0}: Error finding container 04d53fddd58f22bf5db28bf8a60c022cf96181705ca36ba1c9f47c47e8100a25: Status 404 returned error can't find the container with id 04d53fddd58f22bf5db28bf8a60c022cf96181705ca36ba1c9f47c47e8100a25 Jul 27 23:44:57.522354 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:57.522322 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xv85c" event={"ID":"ecc528bb-fcae-4f94-aa61-ce86ce490cf2","Type":"ContainerStarted","Data":"04d53fddd58f22bf5db28bf8a60c022cf96181705ca36ba1c9f47c47e8100a25"} Jul 27 23:44:59.535672 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:59.535639 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" event={"ID":"15412a18-3323-4d49-b43a-6f219f7c977a","Type":"ContainerStarted","Data":"77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad"} Jul 27 23:44:59.536123 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:59.535713 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:44:59.554825 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:44:59.554763 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" podStartSLOduration=1.7670223859999998 podStartE2EDuration="4.554747766s" totalImagesPullingTime="2.78772538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:44:59.552961638 +0000 UTC m=+499.477170517" watchObservedRunningTime="2026-07-27 23:44:59.554747766 +0000 UTC m=+499.478956643" Jul 27 23:45:01.544106 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:01.544075 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xv85c"] Jul 27 23:45:01.547364 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:01.547329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xv85c" event={"ID":"ecc528bb-fcae-4f94-aa61-ce86ce490cf2","Type":"ContainerStarted","Data":"bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d"} Jul 27 23:45:01.566650 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:01.566599 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-xv85c" podStartSLOduration=1.509331653 podStartE2EDuration="5.566583267s" totalImagesPullingTime="4.057251614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:45:01.565642371 +0000 UTC m=+501.489851250" watchObservedRunningTime="2026-07-27 23:45:01.566583267 +0000 UTC m=+501.490792145" Jul 27 23:45:02.552160 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:02.552117 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-xv85c" podUID="ecc528bb-fcae-4f94-aa61-ce86ce490cf2" containerName="authorino" containerID="cri-o://bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d" gracePeriod=30 Jul 27 23:45:02.791335 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:02.791311 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:45:02.896012 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:02.895914 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcs25\" (UniqueName: \"kubernetes.io/projected/ecc528bb-fcae-4f94-aa61-ce86ce490cf2-kube-api-access-vcs25\") pod \"ecc528bb-fcae-4f94-aa61-ce86ce490cf2\" (UID: \"ecc528bb-fcae-4f94-aa61-ce86ce490cf2\") " Jul 27 23:45:02.897993 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:02.897958 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc528bb-fcae-4f94-aa61-ce86ce490cf2-kube-api-access-vcs25" (OuterVolumeSpecName: "kube-api-access-vcs25") pod "ecc528bb-fcae-4f94-aa61-ce86ce490cf2" (UID: "ecc528bb-fcae-4f94-aa61-ce86ce490cf2"). InnerVolumeSpecName "kube-api-access-vcs25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:45:02.996648 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:02.996599 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcs25\" (UniqueName: \"kubernetes.io/projected/ecc528bb-fcae-4f94-aa61-ce86ce490cf2-kube-api-access-vcs25\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:03.557715 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.557681 2555 generic.go:356] "Generic (PLEG): container finished" podID="ecc528bb-fcae-4f94-aa61-ce86ce490cf2" containerID="bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d" exitCode=0 Jul 27 23:45:03.557715 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.557719 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xv85c" Jul 27 23:45:03.558184 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.557729 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xv85c" event={"ID":"ecc528bb-fcae-4f94-aa61-ce86ce490cf2","Type":"ContainerDied","Data":"bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d"} Jul 27 23:45:03.558184 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.557760 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xv85c" event={"ID":"ecc528bb-fcae-4f94-aa61-ce86ce490cf2","Type":"ContainerDied","Data":"04d53fddd58f22bf5db28bf8a60c022cf96181705ca36ba1c9f47c47e8100a25"} Jul 27 23:45:03.558184 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.557777 2555 scope.go:117] "RemoveContainer" containerID="bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d" Jul 27 23:45:03.565905 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.565884 2555 scope.go:117] "RemoveContainer" containerID="bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d" Jul 27 23:45:03.566165 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:03.566144 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d\": container with ID starting with bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d not found: ID does not exist" containerID="bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d" Jul 27 23:45:03.566224 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.566173 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d"} err="failed to get container status \"bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d\": rpc error: code = NotFound desc = could not find container \"bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d\": container with ID starting with bd526320b133d6908db30387bb755740f6460f9765badc23d95caa69b8da991d not found: ID does not exist" Jul 27 23:45:03.576762 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.576740 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xv85c"] Jul 27 23:45:03.580487 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:03.580460 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-xv85c"] Jul 27 23:45:04.565082 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:04.565044 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc528bb-fcae-4f94-aa61-ce86ce490cf2" path="/var/lib/kubelet/pods/ecc528bb-fcae-4f94-aa61-ce86ce490cf2/volumes" Jul 27 23:45:10.541276 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:10.541244 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:45:10.546962 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:10.546925 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:45:10.586527 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:10.586457 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" podUID="15412a18-3323-4d49-b43a-6f219f7c977a" containerName="limitador" containerID="cri-o://77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad" gracePeriod=30 Jul 27 23:45:11.537998 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.537975 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:45:11.591240 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.591208 2555 generic.go:356] "Generic (PLEG): container finished" podID="15412a18-3323-4d49-b43a-6f219f7c977a" containerID="77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad" exitCode=0 Jul 27 23:45:11.591643 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.591249 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" event={"ID":"15412a18-3323-4d49-b43a-6f219f7c977a","Type":"ContainerDied","Data":"77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad"} Jul 27 23:45:11.591643 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.591288 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" event={"ID":"15412a18-3323-4d49-b43a-6f219f7c977a","Type":"ContainerDied","Data":"085b6ecc16330ddfad14754f30d4e8d3c58b3018acde12197a938bcac5ef6ad6"} Jul 27 23:45:11.591643 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.591310 2555 scope.go:117] "RemoveContainer" containerID="77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad" Jul 27 23:45:11.591643 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.591250 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-49dwk" Jul 27 23:45:11.599218 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.599201 2555 scope.go:117] "RemoveContainer" containerID="77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad" Jul 27 23:45:11.599461 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:11.599436 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad\": container with ID starting with 77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad not found: ID does not exist" containerID="77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad" Jul 27 23:45:11.599526 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.599470 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad"} err="failed to get container status \"77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad\": rpc error: code = NotFound desc = could not find container \"77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad\": container with ID starting with 77e12437e5f60485098e17d74b3f643c83a9dba701b1d597e4c7873ac1b4ddad not found: ID does not exist" Jul 27 23:45:11.666799 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.666721 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm76b\" (UniqueName: \"kubernetes.io/projected/15412a18-3323-4d49-b43a-6f219f7c977a-kube-api-access-lm76b\") pod \"15412a18-3323-4d49-b43a-6f219f7c977a\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " Jul 27 23:45:11.666799 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.666784 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/15412a18-3323-4d49-b43a-6f219f7c977a-config-file\") pod \"15412a18-3323-4d49-b43a-6f219f7c977a\" (UID: \"15412a18-3323-4d49-b43a-6f219f7c977a\") " Jul 27 23:45:11.667122 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.667099 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15412a18-3323-4d49-b43a-6f219f7c977a-config-file" (OuterVolumeSpecName: "config-file") pod "15412a18-3323-4d49-b43a-6f219f7c977a" (UID: "15412a18-3323-4d49-b43a-6f219f7c977a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:45:11.668789 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.668769 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15412a18-3323-4d49-b43a-6f219f7c977a-kube-api-access-lm76b" (OuterVolumeSpecName: "kube-api-access-lm76b") pod "15412a18-3323-4d49-b43a-6f219f7c977a" (UID: "15412a18-3323-4d49-b43a-6f219f7c977a"). InnerVolumeSpecName "kube-api-access-lm76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:45:11.767658 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.767621 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm76b\" (UniqueName: \"kubernetes.io/projected/15412a18-3323-4d49-b43a-6f219f7c977a-kube-api-access-lm76b\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:11.767658 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.767654 2555 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/15412a18-3323-4d49-b43a-6f219f7c977a-config-file\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:11.911903 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.911876 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:45:11.916126 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:11.916091 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-49dwk"] Jul 27 23:45:12.564867 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:12.564834 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15412a18-3323-4d49-b43a-6f219f7c977a" path="/var/lib/kubelet/pods/15412a18-3323-4d49-b43a-6f219f7c977a/volumes" Jul 27 23:45:14.657701 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:14.657650 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:14.761028 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:14.760989 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:15.113855 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.113825 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xgz87"] Jul 27 23:45:15.114154 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.114140 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15412a18-3323-4d49-b43a-6f219f7c977a" containerName="limitador" Jul 27 23:45:15.114200 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.114156 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="15412a18-3323-4d49-b43a-6f219f7c977a" containerName="limitador" Jul 27 23:45:15.114200 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.114174 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc528bb-fcae-4f94-aa61-ce86ce490cf2" containerName="authorino" Jul 27 23:45:15.114200 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.114179 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc528bb-fcae-4f94-aa61-ce86ce490cf2" containerName="authorino" Jul 27 23:45:15.114288 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.114236 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc528bb-fcae-4f94-aa61-ce86ce490cf2" containerName="authorino" Jul 27 23:45:15.114288 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.114244 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="15412a18-3323-4d49-b43a-6f219f7c977a" containerName="limitador" Jul 27 23:45:15.118476 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.118455 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.120577 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.120556 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 23:45:15.121182 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.121161 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-djq6g\"" Jul 27 23:45:15.121273 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.121197 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 23:45:15.121330 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.121270 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 23:45:15.124800 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.124778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xgz87"] Jul 27 23:45:15.141536 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.141489 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:15.196625 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.196589 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/712c1751-e6c7-4498-b676-15b59ed93e5a-data\") pod \"postgres-69fb4594cf-xgz87\" (UID: \"712c1751-e6c7-4498-b676-15b59ed93e5a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.196781 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.196729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txh4\" (UniqueName: \"kubernetes.io/projected/712c1751-e6c7-4498-b676-15b59ed93e5a-kube-api-access-6txh4\") pod \"postgres-69fb4594cf-xgz87\" (UID: \"712c1751-e6c7-4498-b676-15b59ed93e5a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.297547 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.297478 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/712c1751-e6c7-4498-b676-15b59ed93e5a-data\") pod \"postgres-69fb4594cf-xgz87\" (UID: \"712c1751-e6c7-4498-b676-15b59ed93e5a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.297721 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.297581 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txh4\" (UniqueName: \"kubernetes.io/projected/712c1751-e6c7-4498-b676-15b59ed93e5a-kube-api-access-6txh4\") pod \"postgres-69fb4594cf-xgz87\" (UID: \"712c1751-e6c7-4498-b676-15b59ed93e5a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.297911 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.297888 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/712c1751-e6c7-4498-b676-15b59ed93e5a-data\") pod \"postgres-69fb4594cf-xgz87\" (UID: \"712c1751-e6c7-4498-b676-15b59ed93e5a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.304615 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.304589 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txh4\" (UniqueName: \"kubernetes.io/projected/712c1751-e6c7-4498-b676-15b59ed93e5a-kube-api-access-6txh4\") pod \"postgres-69fb4594cf-xgz87\" (UID: \"712c1751-e6c7-4498-b676-15b59ed93e5a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.429187 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.429085 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:15.558469 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.558419 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xgz87"] Jul 27 23:45:15.561109 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:45:15.561080 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712c1751_e6c7_4498_b676_15b59ed93e5a.slice/crio-8fee95a732208d35e000693f0b4db2376a61176062f7c12f860d31d31d8cd7b5 WatchSource:0}: Error finding container 8fee95a732208d35e000693f0b4db2376a61176062f7c12f860d31d31d8cd7b5: Status 404 returned error can't find the container with id 8fee95a732208d35e000693f0b4db2376a61176062f7c12f860d31d31d8cd7b5 Jul 27 23:45:15.610946 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:15.610908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" event={"ID":"712c1751-e6c7-4498-b676-15b59ed93e5a","Type":"ContainerStarted","Data":"8fee95a732208d35e000693f0b4db2376a61176062f7c12f860d31d31d8cd7b5"} Jul 27 23:45:20.637529 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:20.637475 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" event={"ID":"712c1751-e6c7-4498-b676-15b59ed93e5a","Type":"ContainerStarted","Data":"241ec42691a7cd72ff66ac759b581c151568dd504dea4223e809610cd16472ed"} Jul 27 23:45:20.637968 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:20.637789 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:20.653690 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:20.653645 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" podStartSLOduration=1.066457203 podStartE2EDuration="5.653631447s" totalImagesPullingTime="4.587174244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:45:20.652547542 +0000 UTC m=+520.576756432" watchObservedRunningTime="2026-07-27 23:45:20.653631447 +0000 UTC m=+520.577840325" Jul 27 23:45:26.670317 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:26.670286 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xgz87" Jul 27 23:45:26.990725 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:26.990687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:27.090961 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.090901 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:27.241249 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.241143 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j4sc9"] Jul 27 23:45:27.247678 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.247656 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:27.248622 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.248600 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j4sc9"] Jul 27 23:45:27.250273 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.250246 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b5mc4\"" Jul 27 23:45:27.408001 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.407960 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvql\" (UniqueName: \"kubernetes.io/projected/c4eac437-94ad-47f0-840a-73d8c1f341e8-kube-api-access-8tvql\") pod \"authorino-5d4b67c446-j4sc9\" (UID: \"c4eac437-94ad-47f0-840a-73d8c1f341e8\") " pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:27.412669 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.412638 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j4sc9"] Jul 27 23:45:27.412885 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:27.412863 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8tvql], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-j4sc9" podUID="c4eac437-94ad-47f0-840a-73d8c1f341e8" Jul 27 23:45:27.437170 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.437137 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-59q5v"] Jul 27 23:45:27.441086 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.441065 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.443858 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.443834 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 27 23:45:27.447081 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.447057 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-59q5v"] Jul 27 23:45:27.487393 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.487354 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-59q5v"] Jul 27 23:45:27.487637 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:27.487613 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-htkh8 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-htkh8 tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-59q5v" podUID="8641d424-0911-4ba9-ae02-7520835a9786" Jul 27 23:45:27.509049 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.508970 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvql\" (UniqueName: \"kubernetes.io/projected/c4eac437-94ad-47f0-840a-73d8c1f341e8-kube-api-access-8tvql\") pod \"authorino-5d4b67c446-j4sc9\" (UID: \"c4eac437-94ad-47f0-840a-73d8c1f341e8\") " pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:27.516343 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.516315 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvql\" (UniqueName: \"kubernetes.io/projected/c4eac437-94ad-47f0-840a-73d8c1f341e8-kube-api-access-8tvql\") pod \"authorino-5d4b67c446-j4sc9\" (UID: \"c4eac437-94ad-47f0-840a-73d8c1f341e8\") " pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:27.610339 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.610301 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkh8\" (UniqueName: \"kubernetes.io/projected/8641d424-0911-4ba9-ae02-7520835a9786-kube-api-access-htkh8\") pod \"authorino-76877994-59q5v\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.610339 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.610340 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8641d424-0911-4ba9-ae02-7520835a9786-tls-cert\") pod \"authorino-76877994-59q5v\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.667064 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.667028 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:27.667234 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.667027 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.674712 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.674689 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:27.677508 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.677473 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.711298 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.711259 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkh8\" (UniqueName: \"kubernetes.io/projected/8641d424-0911-4ba9-ae02-7520835a9786-kube-api-access-htkh8\") pod \"authorino-76877994-59q5v\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.711298 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.711295 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8641d424-0911-4ba9-ae02-7520835a9786-tls-cert\") pod \"authorino-76877994-59q5v\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.713622 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.713599 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8641d424-0911-4ba9-ae02-7520835a9786-tls-cert\") pod \"authorino-76877994-59q5v\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.718182 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.718144 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkh8\" (UniqueName: \"kubernetes.io/projected/8641d424-0911-4ba9-ae02-7520835a9786-kube-api-access-htkh8\") pod \"authorino-76877994-59q5v\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:27.812663 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.812576 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8641d424-0911-4ba9-ae02-7520835a9786-tls-cert\") pod \"8641d424-0911-4ba9-ae02-7520835a9786\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " Jul 27 23:45:27.812663 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.812660 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tvql\" (UniqueName: \"kubernetes.io/projected/c4eac437-94ad-47f0-840a-73d8c1f341e8-kube-api-access-8tvql\") pod \"c4eac437-94ad-47f0-840a-73d8c1f341e8\" (UID: \"c4eac437-94ad-47f0-840a-73d8c1f341e8\") " Jul 27 23:45:27.812840 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.812749 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htkh8\" (UniqueName: \"kubernetes.io/projected/8641d424-0911-4ba9-ae02-7520835a9786-kube-api-access-htkh8\") pod \"8641d424-0911-4ba9-ae02-7520835a9786\" (UID: \"8641d424-0911-4ba9-ae02-7520835a9786\") " Jul 27 23:45:27.814675 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.814641 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8641d424-0911-4ba9-ae02-7520835a9786-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8641d424-0911-4ba9-ae02-7520835a9786" (UID: "8641d424-0911-4ba9-ae02-7520835a9786"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:45:27.814785 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.814682 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eac437-94ad-47f0-840a-73d8c1f341e8-kube-api-access-8tvql" (OuterVolumeSpecName: "kube-api-access-8tvql") pod "c4eac437-94ad-47f0-840a-73d8c1f341e8" (UID: "c4eac437-94ad-47f0-840a-73d8c1f341e8"). InnerVolumeSpecName "kube-api-access-8tvql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:45:27.814785 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.814714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8641d424-0911-4ba9-ae02-7520835a9786-kube-api-access-htkh8" (OuterVolumeSpecName: "kube-api-access-htkh8") pod "8641d424-0911-4ba9-ae02-7520835a9786" (UID: "8641d424-0911-4ba9-ae02-7520835a9786"). InnerVolumeSpecName "kube-api-access-htkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:45:27.913573 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.913539 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvql\" (UniqueName: \"kubernetes.io/projected/c4eac437-94ad-47f0-840a-73d8c1f341e8-kube-api-access-8tvql\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:27.913573 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.913566 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkh8\" (UniqueName: \"kubernetes.io/projected/8641d424-0911-4ba9-ae02-7520835a9786-kube-api-access-htkh8\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:27.913573 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:27.913578 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8641d424-0911-4ba9-ae02-7520835a9786-tls-cert\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:28.670777 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:28.670748 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-j4sc9" Jul 27 23:45:28.670940 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:28.670748 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-59q5v" Jul 27 23:45:28.703113 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:28.703081 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-59q5v"] Jul 27 23:45:28.706442 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:28.706418 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-59q5v"] Jul 27 23:45:28.724957 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:28.724932 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j4sc9"] Jul 27 23:45:28.728536 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:28.728479 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-j4sc9"] Jul 27 23:45:30.566380 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:30.566341 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8641d424-0911-4ba9-ae02-7520835a9786" path="/var/lib/kubelet/pods/8641d424-0911-4ba9-ae02-7520835a9786/volumes" Jul 27 23:45:30.567260 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:30.567237 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4eac437-94ad-47f0-840a-73d8c1f341e8" path="/var/lib/kubelet/pods/c4eac437-94ad-47f0-840a-73d8c1f341e8/volumes" Jul 27 23:45:33.309861 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.309769 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:33.316824 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.316794 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b9b5d4f94-rqlkp"] Jul 27 23:45:33.383568 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.383531 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b9b5d4f94-rqlkp"] Jul 27 23:45:33.383745 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.383591 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:33.385927 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.385902 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 23:45:33.386059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.385924 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x79r4\"" Jul 27 23:45:33.386059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.385907 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 23:45:33.402440 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.402403 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:33.454356 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.454324 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s252g\" (UniqueName: \"kubernetes.io/projected/86598d13-dd5c-46bf-8098-8302f0ce95d5-kube-api-access-s252g\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:33.454356 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.454359 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:33.456175 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.456146 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:33.486747 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.486714 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b8b897757-scz4q"] Jul 27 23:45:33.532323 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.532288 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8b897757-scz4q"] Jul 27 23:45:33.532504 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.532406 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:33.555219 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.555184 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s252g\" (UniqueName: \"kubernetes.io/projected/86598d13-dd5c-46bf-8098-8302f0ce95d5-kube-api-access-s252g\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:33.555219 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.555219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:33.555449 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.555247 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4ll\" (UniqueName: \"kubernetes.io/projected/15510025-9e90-41c5-9fa9-109e52348df4-kube-api-access-nr4ll\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:33.555449 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.555270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:33.555449 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:33.555357 2555 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 27 23:45:33.555449 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:33.555440 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert podName:86598d13-dd5c-46bf-8098-8302f0ce95d5 nodeName:}" failed. No retries permitted until 2026-07-27 23:45:34.055418276 +0000 UTC m=+533.979627134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert") pod "maas-controller-b9b5d4f94-rqlkp" (UID: "86598d13-dd5c-46bf-8098-8302f0ce95d5") : secret "maas-controller-webhook-cert" not found Jul 27 23:45:33.563151 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.563098 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s252g\" (UniqueName: \"kubernetes.io/projected/86598d13-dd5c-46bf-8098-8302f0ce95d5-kube-api-access-s252g\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:33.656480 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.656437 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4ll\" (UniqueName: \"kubernetes.io/projected/15510025-9e90-41c5-9fa9-109e52348df4-kube-api-access-nr4ll\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:33.656714 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.656522 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:33.656714 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:33.656674 2555 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 27 23:45:33.656837 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:33.656735 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert podName:15510025-9e90-41c5-9fa9-109e52348df4 nodeName:}" failed. No retries permitted until 2026-07-27 23:45:34.156715644 +0000 UTC m=+534.080924503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert") pod "maas-controller-7b8b897757-scz4q" (UID: "15510025-9e90-41c5-9fa9-109e52348df4") : secret "maas-controller-webhook-cert" not found Jul 27 23:45:33.666734 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.666707 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4ll\" (UniqueName: \"kubernetes.io/projected/15510025-9e90-41c5-9fa9-109e52348df4-kube-api-access-nr4ll\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:33.795379 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:33.795345 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:34.059721 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.059685 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:34.062233 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.062208 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert\") pod \"maas-controller-b9b5d4f94-rqlkp\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:34.161099 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.161059 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:34.163471 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.163450 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert\") pod \"maas-controller-7b8b897757-scz4q\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:34.294149 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.294109 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:34.434403 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.434377 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b9b5d4f94-rqlkp"] Jul 27 23:45:34.436305 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:45:34.436280 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86598d13_dd5c_46bf_8098_8302f0ce95d5.slice/crio-f337277d9fb15cc3fd1fdf837ac61f8bd6642ad7843982c0bdff731bb3fa6368 WatchSource:0}: Error finding container f337277d9fb15cc3fd1fdf837ac61f8bd6642ad7843982c0bdff731bb3fa6368: Status 404 returned error can't find the container with id f337277d9fb15cc3fd1fdf837ac61f8bd6642ad7843982c0bdff731bb3fa6368 Jul 27 23:45:34.444923 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.444902 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:34.588645 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.588521 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8b897757-scz4q"] Jul 27 23:45:34.591300 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:45:34.591275 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15510025_9e90_41c5_9fa9_109e52348df4.slice/crio-a69ec65e6849eaf2cc023441c7912c33cc872e021e14f8a4e24549e2ea690f8d WatchSource:0}: Error finding container a69ec65e6849eaf2cc023441c7912c33cc872e021e14f8a4e24549e2ea690f8d: Status 404 returned error can't find the container with id a69ec65e6849eaf2cc023441c7912c33cc872e021e14f8a4e24549e2ea690f8d Jul 27 23:45:34.696258 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.696212 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" event={"ID":"86598d13-dd5c-46bf-8098-8302f0ce95d5","Type":"ContainerStarted","Data":"f337277d9fb15cc3fd1fdf837ac61f8bd6642ad7843982c0bdff731bb3fa6368"} Jul 27 23:45:34.697404 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:34.697378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-scz4q" event={"ID":"15510025-9e90-41c5-9fa9-109e52348df4","Type":"ContainerStarted","Data":"a69ec65e6849eaf2cc023441c7912c33cc872e021e14f8a4e24549e2ea690f8d"} Jul 27 23:45:37.314409 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.314375 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:37.429297 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.429209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:37.714611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.714472 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" event={"ID":"86598d13-dd5c-46bf-8098-8302f0ce95d5","Type":"ContainerStarted","Data":"f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b"} Jul 27 23:45:37.714611 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.714559 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:37.716135 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.716099 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-scz4q" event={"ID":"15510025-9e90-41c5-9fa9-109e52348df4","Type":"ContainerStarted","Data":"dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587"} Jul 27 23:45:37.716254 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.716240 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:37.731159 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.731110 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" podStartSLOduration=2.069446519 podStartE2EDuration="4.731091506s" totalImagesPullingTime="2.661644987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:45:37.730140604 +0000 UTC m=+537.654349482" watchObservedRunningTime="2026-07-27 23:45:37.731091506 +0000 UTC m=+537.655300385" Jul 27 23:45:37.748884 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:37.748824 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b8b897757-scz4q" podStartSLOduration=2.252592808 podStartE2EDuration="4.748803122s" totalImagesPullingTime="2.496210314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:45:37.746974861 +0000 UTC m=+537.671183762" watchObservedRunningTime="2026-07-27 23:45:37.748803122 +0000 UTC m=+537.673012001" Jul 27 23:45:48.724016 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:48.723985 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:48.724370 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:48.724079 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:45:48.779681 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:48.779648 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b9b5d4f94-rqlkp"] Jul 27 23:45:48.779866 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:48.779832 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" podUID="86598d13-dd5c-46bf-8098-8302f0ce95d5" containerName="manager" containerID="cri-o://f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b" gracePeriod=30 Jul 27 23:45:49.022870 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.022843 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:49.089472 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.089444 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert\") pod \"86598d13-dd5c-46bf-8098-8302f0ce95d5\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " Jul 27 23:45:49.089673 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.089487 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s252g\" (UniqueName: \"kubernetes.io/projected/86598d13-dd5c-46bf-8098-8302f0ce95d5-kube-api-access-s252g\") pod \"86598d13-dd5c-46bf-8098-8302f0ce95d5\" (UID: \"86598d13-dd5c-46bf-8098-8302f0ce95d5\") " Jul 27 23:45:49.091643 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.091613 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "86598d13-dd5c-46bf-8098-8302f0ce95d5" (UID: "86598d13-dd5c-46bf-8098-8302f0ce95d5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:45:49.091748 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.091706 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86598d13-dd5c-46bf-8098-8302f0ce95d5-kube-api-access-s252g" (OuterVolumeSpecName: "kube-api-access-s252g") pod "86598d13-dd5c-46bf-8098-8302f0ce95d5" (UID: "86598d13-dd5c-46bf-8098-8302f0ce95d5"). InnerVolumeSpecName "kube-api-access-s252g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:45:49.190223 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.190190 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86598d13-dd5c-46bf-8098-8302f0ce95d5-webhook-cert\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:49.190223 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.190218 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s252g\" (UniqueName: \"kubernetes.io/projected/86598d13-dd5c-46bf-8098-8302f0ce95d5-kube-api-access-s252g\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:45:49.768557 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.768524 2555 generic.go:356] "Generic (PLEG): container finished" podID="86598d13-dd5c-46bf-8098-8302f0ce95d5" containerID="f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b" exitCode=0 Jul 27 23:45:49.769042 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.768565 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" Jul 27 23:45:49.769042 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.768601 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" event={"ID":"86598d13-dd5c-46bf-8098-8302f0ce95d5","Type":"ContainerDied","Data":"f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b"} Jul 27 23:45:49.769042 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.768643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b9b5d4f94-rqlkp" event={"ID":"86598d13-dd5c-46bf-8098-8302f0ce95d5","Type":"ContainerDied","Data":"f337277d9fb15cc3fd1fdf837ac61f8bd6642ad7843982c0bdff731bb3fa6368"} Jul 27 23:45:49.769042 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.768665 2555 scope.go:117] "RemoveContainer" containerID="f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b" Jul 27 23:45:49.776855 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.776838 2555 scope.go:117] "RemoveContainer" containerID="f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b" Jul 27 23:45:49.777093 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:45:49.777074 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b\": container with ID starting with f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b not found: ID does not exist" containerID="f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b" Jul 27 23:45:49.777161 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.777100 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b"} err="failed to get container status \"f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b\": rpc error: code = NotFound desc = could not find container \"f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b\": container with ID starting with f093119f38c6ae9a6e467b42dc1440cba6a66eb12befefa5452efd896925ca0b not found: ID does not exist" Jul 27 23:45:49.791335 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.791315 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b9b5d4f94-rqlkp"] Jul 27 23:45:49.796034 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:49.796014 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b9b5d4f94-rqlkp"] Jul 27 23:45:50.564462 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:50.564423 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86598d13-dd5c-46bf-8098-8302f0ce95d5" path="/var/lib/kubelet/pods/86598d13-dd5c-46bf-8098-8302f0ce95d5/volumes" Jul 27 23:45:56.923078 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:56.923035 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:45:57.023059 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:45:57.023026 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:09.063167 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.063131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:09.156193 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.156151 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:09.703826 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.703787 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:09.834118 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.834083 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2"] Jul 27 23:46:09.834481 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.834464 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86598d13-dd5c-46bf-8098-8302f0ce95d5" containerName="manager" Jul 27 23:46:09.834603 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.834484 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86598d13-dd5c-46bf-8098-8302f0ce95d5" containerName="manager" Jul 27 23:46:09.834660 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.834622 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="86598d13-dd5c-46bf-8098-8302f0ce95d5" containerName="manager" Jul 27 23:46:09.838779 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.838758 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:09.841301 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.841277 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rpq2s\"" Jul 27 23:46:09.841301 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.841297 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 27 23:46:09.851044 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.851022 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2"] Jul 27 23:46:09.964536 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.964429 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzq9z\" (UniqueName: \"kubernetes.io/projected/d339da45-e0f5-4608-ba77-966d6d175b6b-kube-api-access-lzq9z\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:09.964536 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.964482 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d339da45-e0f5-4608-ba77-966d6d175b6b-openshift-service-ca\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:09.964704 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:09.964612 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d339da45-e0f5-4608-ba77-966d6d175b6b-maas-api-tls\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.065392 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.065354 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzq9z\" (UniqueName: \"kubernetes.io/projected/d339da45-e0f5-4608-ba77-966d6d175b6b-kube-api-access-lzq9z\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.065799 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.065399 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d339da45-e0f5-4608-ba77-966d6d175b6b-openshift-service-ca\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.065799 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.065442 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d339da45-e0f5-4608-ba77-966d6d175b6b-maas-api-tls\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.065799 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:46:10.065564 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 27 23:46:10.065799 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:46:10.065625 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d339da45-e0f5-4608-ba77-966d6d175b6b-maas-api-tls podName:d339da45-e0f5-4608-ba77-966d6d175b6b nodeName:}" failed. No retries permitted until 2026-07-27 23:46:10.565609657 +0000 UTC m=+570.489818513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d339da45-e0f5-4608-ba77-966d6d175b6b-maas-api-tls") pod "maas-api-655bcd7745-gdkb2" (UID: "d339da45-e0f5-4608-ba77-966d6d175b6b") : secret "maas-api-serving-cert" not found Jul 27 23:46:10.066331 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.066272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d339da45-e0f5-4608-ba77-966d6d175b6b-openshift-service-ca\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.075438 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.075408 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzq9z\" (UniqueName: \"kubernetes.io/projected/d339da45-e0f5-4608-ba77-966d6d175b6b-kube-api-access-lzq9z\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.223802 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.223716 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:10.299855 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.299816 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:10.302816 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.302791 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g"] Jul 27 23:46:10.305267 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.305249 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.307262 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.307248 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 23:46:10.317304 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.317281 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tlfb6"] Jul 27 23:46:10.320193 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.320173 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g"] Jul 27 23:46:10.320333 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.320276 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.322442 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.322425 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rq9m9\"" Jul 27 23:46:10.331487 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.331457 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tlfb6"] Jul 27 23:46:10.470036 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.469996 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba737cb-3def-43c3-92f1-cb217791ff3f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-p7l7g\" (UID: \"0ba737cb-3def-43c3-92f1-cb217791ff3f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.470036 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.470040 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fpv\" (UniqueName: \"kubernetes.io/projected/0ba737cb-3def-43c3-92f1-cb217791ff3f-kube-api-access-d7fpv\") pod \"payload-pre-processing-687dbb6cf6-p7l7g\" (UID: \"0ba737cb-3def-43c3-92f1-cb217791ff3f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.470246 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.470068 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzgx\" (UniqueName: \"kubernetes.io/projected/cb094531-c0c3-4729-a052-0b2cd6e835d2-kube-api-access-qwzgx\") pod \"payload-processing-5cc6cd444b-tlfb6\" (UID: \"cb094531-c0c3-4729-a052-0b2cd6e835d2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.470246 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.470112 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb094531-c0c3-4729-a052-0b2cd6e835d2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tlfb6\" (UID: \"cb094531-c0c3-4729-a052-0b2cd6e835d2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.571348 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.571318 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzgx\" (UniqueName: \"kubernetes.io/projected/cb094531-c0c3-4729-a052-0b2cd6e835d2-kube-api-access-qwzgx\") pod \"payload-processing-5cc6cd444b-tlfb6\" (UID: \"cb094531-c0c3-4729-a052-0b2cd6e835d2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.571558 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.571371 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb094531-c0c3-4729-a052-0b2cd6e835d2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tlfb6\" (UID: \"cb094531-c0c3-4729-a052-0b2cd6e835d2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.571558 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.571398 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d339da45-e0f5-4608-ba77-966d6d175b6b-maas-api-tls\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.571558 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.571433 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba737cb-3def-43c3-92f1-cb217791ff3f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-p7l7g\" (UID: \"0ba737cb-3def-43c3-92f1-cb217791ff3f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.571558 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.571452 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fpv\" (UniqueName: \"kubernetes.io/projected/0ba737cb-3def-43c3-92f1-cb217791ff3f-kube-api-access-d7fpv\") pod \"payload-pre-processing-687dbb6cf6-p7l7g\" (UID: \"0ba737cb-3def-43c3-92f1-cb217791ff3f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.572159 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.572133 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba737cb-3def-43c3-92f1-cb217791ff3f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-p7l7g\" (UID: \"0ba737cb-3def-43c3-92f1-cb217791ff3f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.572257 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.572143 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb094531-c0c3-4729-a052-0b2cd6e835d2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tlfb6\" (UID: \"cb094531-c0c3-4729-a052-0b2cd6e835d2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.573808 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.573789 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d339da45-e0f5-4608-ba77-966d6d175b6b-maas-api-tls\") pod \"maas-api-655bcd7745-gdkb2\" (UID: \"d339da45-e0f5-4608-ba77-966d6d175b6b\") " pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.578800 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.578779 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzgx\" (UniqueName: \"kubernetes.io/projected/cb094531-c0c3-4729-a052-0b2cd6e835d2-kube-api-access-qwzgx\") pod \"payload-processing-5cc6cd444b-tlfb6\" (UID: \"cb094531-c0c3-4729-a052-0b2cd6e835d2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.579120 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.579100 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fpv\" (UniqueName: \"kubernetes.io/projected/0ba737cb-3def-43c3-92f1-cb217791ff3f-kube-api-access-d7fpv\") pod \"payload-pre-processing-687dbb6cf6-p7l7g\" (UID: \"0ba737cb-3def-43c3-92f1-cb217791ff3f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.616163 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.616133 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:10.633582 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.633554 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:10.748214 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.748177 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:10.761194 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.760768 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g"] Jul 27 23:46:10.763985 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:46:10.763954 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba737cb_3def_43c3_92f1_cb217791ff3f.slice/crio-3e28c1b71849a4e8525ec67241c2e211e1f2a7e890f67332f5c74a9e344eba24 WatchSource:0}: Error finding container 3e28c1b71849a4e8525ec67241c2e211e1f2a7e890f67332f5c74a9e344eba24: Status 404 returned error can't find the container with id 3e28c1b71849a4e8525ec67241c2e211e1f2a7e890f67332f5c74a9e344eba24 Jul 27 23:46:10.793163 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.793136 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tlfb6"] Jul 27 23:46:10.795117 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:46:10.795085 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb094531_c0c3_4729_a052_0b2cd6e835d2.slice/crio-40bab63595ade7c728214602b443099243ce7075367e64ae991901d092df2bbf WatchSource:0}: Error finding container 40bab63595ade7c728214602b443099243ce7075367e64ae991901d092df2bbf: Status 404 returned error can't find the container with id 40bab63595ade7c728214602b443099243ce7075367e64ae991901d092df2bbf Jul 27 23:46:10.862190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.862154 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" event={"ID":"cb094531-c0c3-4729-a052-0b2cd6e835d2","Type":"ContainerStarted","Data":"40bab63595ade7c728214602b443099243ce7075367e64ae991901d092df2bbf"} Jul 27 23:46:10.863552 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.863520 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" event={"ID":"0ba737cb-3def-43c3-92f1-cb217791ff3f","Type":"ContainerStarted","Data":"3e28c1b71849a4e8525ec67241c2e211e1f2a7e890f67332f5c74a9e344eba24"} Jul 27 23:46:10.887404 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:10.887381 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2"] Jul 27 23:46:10.890014 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:46:10.889990 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd339da45_e0f5_4608_ba77_966d6d175b6b.slice/crio-fdd4512099bb3328c2631b3b6ddf15b1ba1139b6082fdcf35c6d8f1f077e8d97 WatchSource:0}: Error finding container fdd4512099bb3328c2631b3b6ddf15b1ba1139b6082fdcf35c6d8f1f077e8d97: Status 404 returned error can't find the container with id fdd4512099bb3328c2631b3b6ddf15b1ba1139b6082fdcf35c6d8f1f077e8d97 Jul 27 23:46:11.890811 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:11.890772 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" event={"ID":"d339da45-e0f5-4608-ba77-966d6d175b6b","Type":"ContainerStarted","Data":"fdd4512099bb3328c2631b3b6ddf15b1ba1139b6082fdcf35c6d8f1f077e8d97"} Jul 27 23:46:13.904276 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.904189 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" event={"ID":"cb094531-c0c3-4729-a052-0b2cd6e835d2","Type":"ContainerStarted","Data":"8bdae56bd0d339bd160ff087ba83fc7233f06f630a05e7919a7bf2caded69c0c"} Jul 27 23:46:13.904730 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.904323 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:13.905933 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.905900 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" event={"ID":"0ba737cb-3def-43c3-92f1-cb217791ff3f","Type":"ContainerStarted","Data":"1f5e6564fe02d877aaa673f7cf62d055c075f46c7c4c44bed45c4fdd3fe366f7"} Jul 27 23:46:13.906067 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.905968 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:13.907642 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.907617 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" event={"ID":"d339da45-e0f5-4608-ba77-966d6d175b6b","Type":"ContainerStarted","Data":"924150243af6c979408e9f25e24248f45a63d94a439dfa9408d20f70045c524e"} Jul 27 23:46:13.907755 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.907674 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:13.918664 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.918621 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" podStartSLOduration=1.059273772 podStartE2EDuration="3.918607787s" totalImagesPullingTime="2.859334015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:13.917443921 +0000 UTC m=+573.841652797" watchObservedRunningTime="2026-07-27 23:46:13.918607787 +0000 UTC m=+573.842816643" Jul 27 23:46:13.931042 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.930996 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" podStartSLOduration=1.039682684 podStartE2EDuration="3.930982685s" totalImagesPullingTime="2.891300001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:13.93000289 +0000 UTC m=+573.854211768" watchObservedRunningTime="2026-07-27 23:46:13.930982685 +0000 UTC m=+573.855191562" Jul 27 23:46:13.944440 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:13.944393 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" podStartSLOduration=2.208366798 podStartE2EDuration="4.944379554s" totalImagesPullingTime="2.736012756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:13.943252604 +0000 UTC m=+573.867461483" watchObservedRunningTime="2026-07-27 23:46:13.944379554 +0000 UTC m=+573.868588432" Jul 27 23:46:19.920152 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:19.920123 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-655bcd7745-gdkb2" Jul 27 23:46:20.912278 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:20.912236 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:21.010399 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:21.010350 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:24.916931 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:24.916899 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-p7l7g" Jul 27 23:46:24.917304 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:24.916986 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-tlfb6" Jul 27 23:46:25.931931 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:25.931890 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:31.272301 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:31.272264 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:32.599797 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:32.599757 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:39.248528 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.248476 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl"] Jul 27 23:46:39.250891 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.250872 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.253636 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.253615 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 23:46:39.253721 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.253616 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 27 23:46:39.253792 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.253617 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kmtzt\"" Jul 27 23:46:39.253792 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.253621 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 23:46:39.259085 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.259059 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl"] Jul 27 23:46:39.320769 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.320736 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.320928 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.320780 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.320928 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.320812 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.320928 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.320868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.320928 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.320895 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnfx\" (UniqueName: \"kubernetes.io/projected/4565b4a3-734a-4376-934f-54ad84f5e32e-kube-api-access-nbnfx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.321064 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.320938 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565b4a3-734a-4376-934f-54ad84f5e32e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.326057 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.326031 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:39.421632 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.421804 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421643 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.421804 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421681 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.421804 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421709 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.421804 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421725 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnfx\" (UniqueName: \"kubernetes.io/projected/4565b4a3-734a-4376-934f-54ad84f5e32e-kube-api-access-nbnfx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.421804 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421753 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565b4a3-734a-4376-934f-54ad84f5e32e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.422048 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.421999 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.422125 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.422104 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.422167 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.422147 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.424068 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.424043 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4565b4a3-734a-4376-934f-54ad84f5e32e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.424208 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.424189 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4565b4a3-734a-4376-934f-54ad84f5e32e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.428732 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.428708 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnfx\" (UniqueName: \"kubernetes.io/projected/4565b4a3-734a-4376-934f-54ad84f5e32e-kube-api-access-nbnfx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6hfjl\" (UID: \"4565b4a3-734a-4376-934f-54ad84f5e32e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.562217 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.562129 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:39.690663 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:39.690636 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl"] Jul 27 23:46:39.692758 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:46:39.692725 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4565b4a3_734a_4376_934f_54ad84f5e32e.slice/crio-b59c40da21eb94b7ccd740d32f3f3855bcd4bc23b85bc90e9d852ecd7bc7cef5 WatchSource:0}: Error finding container b59c40da21eb94b7ccd740d32f3f3855bcd4bc23b85bc90e9d852ecd7bc7cef5: Status 404 returned error can't find the container with id b59c40da21eb94b7ccd740d32f3f3855bcd4bc23b85bc90e9d852ecd7bc7cef5 Jul 27 23:46:40.031151 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:40.031108 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" event={"ID":"4565b4a3-734a-4376-934f-54ad84f5e32e","Type":"ContainerStarted","Data":"b59c40da21eb94b7ccd740d32f3f3855bcd4bc23b85bc90e9d852ecd7bc7cef5"} Jul 27 23:46:41.036669 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:41.036632 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" event={"ID":"4565b4a3-734a-4376-934f-54ad84f5e32e","Type":"ContainerStarted","Data":"7c2676af309fcda7e5d0a0817e355c68d81f90593e8bceef079d2a932e542100"} Jul 27 23:46:41.054935 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:41.054880 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" podStartSLOduration=1.070339485 podStartE2EDuration="2.05486412s" totalImagesPullingTime="984.524635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:41.05403247 +0000 UTC m=+600.978241347" watchObservedRunningTime="2026-07-27 23:46:41.05486412 +0000 UTC m=+600.979072998" Jul 27 23:46:49.563137 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:49.563099 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:49.564583 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:49.563151 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:49.575295 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:49.575274 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:50.086608 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:50.086577 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6hfjl" Jul 27 23:46:54.629166 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.629132 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw"] Jul 27 23:46:54.632790 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.632765 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.634999 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.634966 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 27 23:46:54.649462 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.649415 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw"] Jul 27 23:46:54.656506 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.656451 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:54.754244 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.754203 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.754244 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.754247 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.754455 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.754340 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.754455 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.754395 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.754455 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.754420 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec627b61-b918-47d3-9f97-49a8ce08d6d7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.754588 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.754456 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qbn\" (UniqueName: \"kubernetes.io/projected/ec627b61-b918-47d3-9f97-49a8ce08d6d7-kube-api-access-s5qbn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.854989 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.854952 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.854989 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.854988 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855219 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855029 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855219 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855219 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855075 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec627b61-b918-47d3-9f97-49a8ce08d6d7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855219 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855092 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qbn\" (UniqueName: \"kubernetes.io/projected/ec627b61-b918-47d3-9f97-49a8ce08d6d7-kube-api-access-s5qbn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855430 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855403 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855522 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855448 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.855589 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.855566 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.857575 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.857546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec627b61-b918-47d3-9f97-49a8ce08d6d7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.857712 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.857695 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec627b61-b918-47d3-9f97-49a8ce08d6d7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.862714 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.862692 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qbn\" (UniqueName: \"kubernetes.io/projected/ec627b61-b918-47d3-9f97-49a8ce08d6d7-kube-api-access-s5qbn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw\" (UID: \"ec627b61-b918-47d3-9f97-49a8ce08d6d7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:54.943486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:54.943400 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:46:55.073919 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:55.073884 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw"] Jul 27 23:46:55.075689 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:46:55.075663 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec627b61_b918_47d3_9f97_49a8ce08d6d7.slice/crio-8cc62b2699077974bbb714376bd48ff52a0df679d3bb589aa569a99b9a093bbe WatchSource:0}: Error finding container 8cc62b2699077974bbb714376bd48ff52a0df679d3bb589aa569a99b9a093bbe: Status 404 returned error can't find the container with id 8cc62b2699077974bbb714376bd48ff52a0df679d3bb589aa569a99b9a093bbe Jul 27 23:46:55.076922 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:55.076905 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 23:46:55.104409 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:55.104367 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" event={"ID":"ec627b61-b918-47d3-9f97-49a8ce08d6d7","Type":"ContainerStarted","Data":"8cc62b2699077974bbb714376bd48ff52a0df679d3bb589aa569a99b9a093bbe"} Jul 27 23:46:56.110197 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:56.110159 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" event={"ID":"ec627b61-b918-47d3-9f97-49a8ce08d6d7","Type":"ContainerStarted","Data":"7498cc31cab980c0d81b956fa9e64db947a516e253ff814e41c1b26480c63ca4"} Jul 27 23:46:56.126040 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:46:56.125991 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" podStartSLOduration=1.943181697 podStartE2EDuration="2.12597423s" totalImagesPullingTime="182.792533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:56.125032222 +0000 UTC m=+616.049241099" watchObservedRunningTime="2026-07-27 23:46:56.12597423 +0000 UTC m=+616.050183108" Jul 27 23:47:02.334774 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.334740 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn"] Jul 27 23:47:02.338109 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.338082 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.340295 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.340270 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 23:47:02.349215 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.349189 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn"] Jul 27 23:47:02.361191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.361149 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:02.421407 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.421364 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.421616 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.421414 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.421616 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.421459 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.421616 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.421542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.421616 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.421566 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfhj\" (UniqueName: \"kubernetes.io/projected/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-kube-api-access-lpfhj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.421616 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.421597 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.522670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.522627 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.522670 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.522672 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.522894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.522698 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.522894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.522717 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfhj\" (UniqueName: \"kubernetes.io/projected/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-kube-api-access-lpfhj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.522894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.522758 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.522894 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.522836 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.523102 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.523079 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.523154 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.523136 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.523190 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.523171 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.524980 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.524953 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.525237 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.525220 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.530551 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.530529 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfhj\" (UniqueName: \"kubernetes.io/projected/69aaa479-b1a6-40f9-bf5c-9eaac60c7816-kube-api-access-lpfhj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l42pn\" (UID: \"69aaa479-b1a6-40f9-bf5c-9eaac60c7816\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.649734 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.649645 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:02.787993 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:02.785854 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn"] Jul 27 23:47:03.143742 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:03.143707 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" event={"ID":"69aaa479-b1a6-40f9-bf5c-9eaac60c7816","Type":"ContainerStarted","Data":"2ac80e94881ad3b79b112d2438bf8bd1aa4fabc5cb535284f4eac0ed7accfcd3"} Jul 27 23:47:03.143902 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:03.143773 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" event={"ID":"69aaa479-b1a6-40f9-bf5c-9eaac60c7816","Type":"ContainerStarted","Data":"153abd6d7268740f0a15f8eb356d159ea181931b45dfcca57eb6eefebced2474"} Jul 27 23:47:03.160962 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:03.160912 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" podStartSLOduration=0.94054748 podStartE2EDuration="1.160897532s" totalImagesPullingTime="220.350052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:03.159369333 +0000 UTC m=+623.083578211" watchObservedRunningTime="2026-07-27 23:47:03.160897532 +0000 UTC m=+623.085106442" Jul 27 23:47:04.944464 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:04.944429 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:47:04.944890 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:04.944526 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:47:04.956813 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:04.956784 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:47:05.163908 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:05.163876 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw" Jul 27 23:47:10.581698 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:10.581661 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:12.650132 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:12.650097 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:12.650625 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:12.650157 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:12.662936 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:12.662906 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:13.202850 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:13.202815 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l42pn" Jul 27 23:47:24.616544 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:24.616487 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:55.446083 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:47:55.446049 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:06.645417 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:48:06.645382 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:35.048348 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:48:35.048251 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:35.153133 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:48:35.153098 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:36.169143 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:48:36.169101 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:36.267724 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:48:36.267684 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:22.417431 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.417395 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8b897757-scz4q"] Jul 27 23:49:22.417903 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.417616 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b8b897757-scz4q" podUID="15510025-9e90-41c5-9fa9-109e52348df4" containerName="manager" containerID="cri-o://dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587" gracePeriod=30 Jul 27 23:49:22.676056 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.675998 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:49:22.721483 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.721453 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr4ll\" (UniqueName: \"kubernetes.io/projected/15510025-9e90-41c5-9fa9-109e52348df4-kube-api-access-nr4ll\") pod \"15510025-9e90-41c5-9fa9-109e52348df4\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " Jul 27 23:49:22.721647 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.721514 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert\") pod \"15510025-9e90-41c5-9fa9-109e52348df4\" (UID: \"15510025-9e90-41c5-9fa9-109e52348df4\") " Jul 27 23:49:22.723557 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.723532 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "15510025-9e90-41c5-9fa9-109e52348df4" (UID: "15510025-9e90-41c5-9fa9-109e52348df4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:49:22.723679 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.723573 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15510025-9e90-41c5-9fa9-109e52348df4-kube-api-access-nr4ll" (OuterVolumeSpecName: "kube-api-access-nr4ll") pod "15510025-9e90-41c5-9fa9-109e52348df4" (UID: "15510025-9e90-41c5-9fa9-109e52348df4"). InnerVolumeSpecName "kube-api-access-nr4ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:49:22.773256 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.773227 2555 generic.go:356] "Generic (PLEG): container finished" podID="15510025-9e90-41c5-9fa9-109e52348df4" containerID="dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587" exitCode=0 Jul 27 23:49:22.773408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.773276 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-scz4q" Jul 27 23:49:22.773408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.773314 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-scz4q" event={"ID":"15510025-9e90-41c5-9fa9-109e52348df4","Type":"ContainerDied","Data":"dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587"} Jul 27 23:49:22.773408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.773370 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-scz4q" event={"ID":"15510025-9e90-41c5-9fa9-109e52348df4","Type":"ContainerDied","Data":"a69ec65e6849eaf2cc023441c7912c33cc872e021e14f8a4e24549e2ea690f8d"} Jul 27 23:49:22.773408 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.773398 2555 scope.go:117] "RemoveContainer" containerID="dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587" Jul 27 23:49:22.781882 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.781866 2555 scope.go:117] "RemoveContainer" containerID="dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587" Jul 27 23:49:22.782159 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:49:22.782137 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587\": container with ID starting with dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587 not found: ID does not exist" containerID="dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587" Jul 27 23:49:22.782211 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.782167 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587"} err="failed to get container status \"dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587\": rpc error: code = NotFound desc = could not find container \"dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587\": container with ID starting with dc535f9a2061745e161d01ce605cc252cc73cb20ad780f324922e135d8a95587 not found: ID does not exist" Jul 27 23:49:22.792566 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.792544 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8b897757-scz4q"] Jul 27 23:49:22.794994 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.794973 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b8b897757-scz4q"] Jul 27 23:49:22.822121 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.822094 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr4ll\" (UniqueName: \"kubernetes.io/projected/15510025-9e90-41c5-9fa9-109e52348df4-kube-api-access-nr4ll\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:49:22.822121 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:22.822116 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15510025-9e90-41c5-9fa9-109e52348df4-webhook-cert\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:49:24.565101 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:49:24.565067 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15510025-9e90-41c5-9fa9-109e52348df4" path="/var/lib/kubelet/pods/15510025-9e90-41c5-9fa9-109e52348df4/volumes" Jul 27 23:50:23.592529 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.592477 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b8b897757-6ndmz"] Jul 27 23:50:23.593102 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.593014 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15510025-9e90-41c5-9fa9-109e52348df4" containerName="manager" Jul 27 23:50:23.593102 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.593029 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="15510025-9e90-41c5-9fa9-109e52348df4" containerName="manager" Jul 27 23:50:23.593194 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.593141 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="15510025-9e90-41c5-9fa9-109e52348df4" containerName="manager" Jul 27 23:50:23.596383 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.596365 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.599460 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.599432 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 23:50:23.599460 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.599437 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x79r4\"" Jul 27 23:50:23.599460 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.599444 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 23:50:23.605511 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.605477 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8b897757-6ndmz"] Jul 27 23:50:23.650075 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.650048 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4fl\" (UniqueName: \"kubernetes.io/projected/cdf7fefa-50dd-47d0-8128-ca021fead3ef-kube-api-access-8n4fl\") pod \"maas-controller-7b8b897757-6ndmz\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.650189 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.650122 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf7fefa-50dd-47d0-8128-ca021fead3ef-webhook-cert\") pod \"maas-controller-7b8b897757-6ndmz\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.750820 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.750785 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf7fefa-50dd-47d0-8128-ca021fead3ef-webhook-cert\") pod \"maas-controller-7b8b897757-6ndmz\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.750960 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.750930 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4fl\" (UniqueName: \"kubernetes.io/projected/cdf7fefa-50dd-47d0-8128-ca021fead3ef-kube-api-access-8n4fl\") pod \"maas-controller-7b8b897757-6ndmz\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.753254 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.753231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf7fefa-50dd-47d0-8128-ca021fead3ef-webhook-cert\") pod \"maas-controller-7b8b897757-6ndmz\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.758775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.758756 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4fl\" (UniqueName: \"kubernetes.io/projected/cdf7fefa-50dd-47d0-8128-ca021fead3ef-kube-api-access-8n4fl\") pod \"maas-controller-7b8b897757-6ndmz\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:23.908206 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:23.908125 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:24.035321 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:24.035295 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8b897757-6ndmz"] Jul 27 23:50:24.036762 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:50:24.036734 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf7fefa_50dd_47d0_8128_ca021fead3ef.slice/crio-ba8eb1e16a2e5e7255af7ef0efcb9c51b7ec856bfc2dca788d1a971a40d04e7e WatchSource:0}: Error finding container ba8eb1e16a2e5e7255af7ef0efcb9c51b7ec856bfc2dca788d1a971a40d04e7e: Status 404 returned error can't find the container with id ba8eb1e16a2e5e7255af7ef0efcb9c51b7ec856bfc2dca788d1a971a40d04e7e Jul 27 23:50:24.056832 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:24.056803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-6ndmz" event={"ID":"cdf7fefa-50dd-47d0-8128-ca021fead3ef","Type":"ContainerStarted","Data":"ba8eb1e16a2e5e7255af7ef0efcb9c51b7ec856bfc2dca788d1a971a40d04e7e"} Jul 27 23:50:25.063719 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:25.063683 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-6ndmz" event={"ID":"cdf7fefa-50dd-47d0-8128-ca021fead3ef","Type":"ContainerStarted","Data":"40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59"} Jul 27 23:50:25.064112 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:25.063812 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:25.079789 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:25.079740 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b8b897757-6ndmz" podStartSLOduration=1.699074645 podStartE2EDuration="2.079725616s" totalImagesPullingTime="380.650971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:50:25.07851771 +0000 UTC m=+825.002726583" watchObservedRunningTime="2026-07-27 23:50:25.079725616 +0000 UTC m=+825.003934525" Jul 27 23:50:36.070839 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:36.070810 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:40.643547 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:40.643513 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:40.721910 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:40.721879 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8b897757-6ndmz"] Jul 27 23:50:40.722097 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:40.722073 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b8b897757-6ndmz" podUID="cdf7fefa-50dd-47d0-8128-ca021fead3ef" containerName="manager" containerID="cri-o://40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59" gracePeriod=30 Jul 27 23:50:40.969159 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:40.969130 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:41.104138 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.104104 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf7fefa-50dd-47d0-8128-ca021fead3ef-webhook-cert\") pod \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " Jul 27 23:50:41.104331 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.104172 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n4fl\" (UniqueName: \"kubernetes.io/projected/cdf7fefa-50dd-47d0-8128-ca021fead3ef-kube-api-access-8n4fl\") pod \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\" (UID: \"cdf7fefa-50dd-47d0-8128-ca021fead3ef\") " Jul 27 23:50:41.106069 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.106030 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf7fefa-50dd-47d0-8128-ca021fead3ef-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cdf7fefa-50dd-47d0-8128-ca021fead3ef" (UID: "cdf7fefa-50dd-47d0-8128-ca021fead3ef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:50:41.106183 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.106122 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf7fefa-50dd-47d0-8128-ca021fead3ef-kube-api-access-8n4fl" (OuterVolumeSpecName: "kube-api-access-8n4fl") pod "cdf7fefa-50dd-47d0-8128-ca021fead3ef" (UID: "cdf7fefa-50dd-47d0-8128-ca021fead3ef"). InnerVolumeSpecName "kube-api-access-8n4fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:50:41.139442 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.139403 2555 generic.go:356] "Generic (PLEG): container finished" podID="cdf7fefa-50dd-47d0-8128-ca021fead3ef" containerID="40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59" exitCode=0 Jul 27 23:50:41.139442 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.139435 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-6ndmz" event={"ID":"cdf7fefa-50dd-47d0-8128-ca021fead3ef","Type":"ContainerDied","Data":"40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59"} Jul 27 23:50:41.139637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.139453 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-6ndmz" Jul 27 23:50:41.139637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.139505 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-6ndmz" event={"ID":"cdf7fefa-50dd-47d0-8128-ca021fead3ef","Type":"ContainerDied","Data":"ba8eb1e16a2e5e7255af7ef0efcb9c51b7ec856bfc2dca788d1a971a40d04e7e"} Jul 27 23:50:41.139637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.139523 2555 scope.go:117] "RemoveContainer" containerID="40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59" Jul 27 23:50:41.148666 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.148649 2555 scope.go:117] "RemoveContainer" containerID="40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59" Jul 27 23:50:41.148911 ip-10-0-130-94 kubenswrapper[2555]: E0727 23:50:41.148891 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59\": container with ID starting with 40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59 not found: ID does not exist" containerID="40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59" Jul 27 23:50:41.148961 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.148920 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59"} err="failed to get container status \"40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59\": rpc error: code = NotFound desc = could not find container \"40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59\": container with ID starting with 40ecfc3f8d280e9b7bc8f5e07a0f55d8f3cea3165f085455bd155d6a6aad6f59 not found: ID does not exist" Jul 27 23:50:41.160063 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.160043 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8b897757-6ndmz"] Jul 27 23:50:41.163892 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.163871 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b8b897757-6ndmz"] Jul 27 23:50:41.205479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.205413 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4fl\" (UniqueName: \"kubernetes.io/projected/cdf7fefa-50dd-47d0-8128-ca021fead3ef-kube-api-access-8n4fl\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:50:41.205479 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.205440 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf7fefa-50dd-47d0-8128-ca021fead3ef-webhook-cert\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:50:41.850212 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.850175 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:41.960063 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.960031 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b8b897757-q8hm7"] Jul 27 23:50:41.960410 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.960398 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf7fefa-50dd-47d0-8128-ca021fead3ef" containerName="manager" Jul 27 23:50:41.960457 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.960413 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf7fefa-50dd-47d0-8128-ca021fead3ef" containerName="manager" Jul 27 23:50:41.960530 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.960473 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf7fefa-50dd-47d0-8128-ca021fead3ef" containerName="manager" Jul 27 23:50:41.964454 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.964434 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:41.966909 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.966885 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 23:50:41.966989 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.966885 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 23:50:41.966989 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.966890 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x79r4\"" Jul 27 23:50:41.972044 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:41.972019 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8b897757-q8hm7"] Jul 27 23:50:42.114427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.114334 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a-webhook-cert\") pod \"maas-controller-7b8b897757-q8hm7\" (UID: \"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a\") " pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.114427 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.114375 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wp4\" (UniqueName: \"kubernetes.io/projected/ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a-kube-api-access-m4wp4\") pod \"maas-controller-7b8b897757-q8hm7\" (UID: \"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a\") " pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.215320 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.215289 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wp4\" (UniqueName: \"kubernetes.io/projected/ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a-kube-api-access-m4wp4\") pod \"maas-controller-7b8b897757-q8hm7\" (UID: \"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a\") " pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.215514 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.215386 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a-webhook-cert\") pod \"maas-controller-7b8b897757-q8hm7\" (UID: \"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a\") " pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.217705 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.217675 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a-webhook-cert\") pod \"maas-controller-7b8b897757-q8hm7\" (UID: \"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a\") " pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.223384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.223363 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wp4\" (UniqueName: \"kubernetes.io/projected/ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a-kube-api-access-m4wp4\") pod \"maas-controller-7b8b897757-q8hm7\" (UID: \"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a\") " pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.276022 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.275993 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:42.404641 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.404613 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8b897757-q8hm7"] Jul 27 23:50:42.406673 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:50:42.406644 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7d0a8c_7c2a_4fbc_bd1c_0da3ae14ee4a.slice/crio-779f282f4c1b964d587d640b8d1171c816795d97a648f2802a1704068feea334 WatchSource:0}: Error finding container 779f282f4c1b964d587d640b8d1171c816795d97a648f2802a1704068feea334: Status 404 returned error can't find the container with id 779f282f4c1b964d587d640b8d1171c816795d97a648f2802a1704068feea334 Jul 27 23:50:42.564900 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:42.564864 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf7fefa-50dd-47d0-8128-ca021fead3ef" path="/var/lib/kubelet/pods/cdf7fefa-50dd-47d0-8128-ca021fead3ef/volumes" Jul 27 23:50:43.151784 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:43.151746 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-q8hm7" event={"ID":"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a","Type":"ContainerStarted","Data":"04fe3c93dfdf1bb030700e5fc1699fb0370ba3e2901a46cb17c8da0341411fa3"} Jul 27 23:50:43.152177 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:43.151848 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8b897757-q8hm7" event={"ID":"ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a","Type":"ContainerStarted","Data":"779f282f4c1b964d587d640b8d1171c816795d97a648f2802a1704068feea334"} Jul 27 23:50:43.152177 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:43.151976 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:50:43.168373 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:43.168318 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b8b897757-q8hm7" podStartSLOduration=1.501430327 podStartE2EDuration="2.168296871s" totalImagesPullingTime="666.866544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:50:43.165839561 +0000 UTC m=+843.090048439" watchObservedRunningTime="2026-07-27 23:50:43.168296871 +0000 UTC m=+843.092505749" Jul 27 23:50:54.159803 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:50:54.159771 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8b897757-q8hm7" Jul 27 23:51:16.605105 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:16.605072 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:16.702882 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:16.702833 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:16.895347 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:16.895246 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:22.369377 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:22.369336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:22.431311 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:22.431262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:22.517761 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:22.517724 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:22.617701 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:22.617667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:22.773054 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:22.773017 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:28.350337 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:28.350300 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:28.446273 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:51:28.446237 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:39.526366 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:52:39.526330 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:48.527471 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:52:48.527426 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:48.583012 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:52:48.582967 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:50.624760 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:52:50.624714 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:50.677671 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:52:50.677637 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:28.283623 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:28.283589 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:32.678918 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:32.678880 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl"] Jul 27 23:55:32.679357 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:32.679093 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" podUID="1463f956-55d1-4d41-8f37-17cfc2de87c8" containerName="manager" containerID="cri-o://58dffd0cf74f8f6552b8e73f35482c19e637018af31afb5b23ae597e69c5fbee" gracePeriod=10 Jul 27 23:55:36.504153 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.504122 2555 generic.go:356] "Generic (PLEG): container finished" podID="1463f956-55d1-4d41-8f37-17cfc2de87c8" containerID="58dffd0cf74f8f6552b8e73f35482c19e637018af31afb5b23ae597e69c5fbee" exitCode=0 Jul 27 23:55:36.504566 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.504161 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" event={"ID":"1463f956-55d1-4d41-8f37-17cfc2de87c8","Type":"ContainerDied","Data":"58dffd0cf74f8f6552b8e73f35482c19e637018af31afb5b23ae597e69c5fbee"} Jul 27 23:55:36.833922 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.833896 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:55:36.987104 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.987070 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1463f956-55d1-4d41-8f37-17cfc2de87c8-extensions-socket-volume\") pod \"1463f956-55d1-4d41-8f37-17cfc2de87c8\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " Jul 27 23:55:36.987276 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.987187 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c6pf\" (UniqueName: \"kubernetes.io/projected/1463f956-55d1-4d41-8f37-17cfc2de87c8-kube-api-access-8c6pf\") pod \"1463f956-55d1-4d41-8f37-17cfc2de87c8\" (UID: \"1463f956-55d1-4d41-8f37-17cfc2de87c8\") " Jul 27 23:55:36.987434 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.987409 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1463f956-55d1-4d41-8f37-17cfc2de87c8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1463f956-55d1-4d41-8f37-17cfc2de87c8" (UID: "1463f956-55d1-4d41-8f37-17cfc2de87c8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:55:36.989152 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:36.989128 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1463f956-55d1-4d41-8f37-17cfc2de87c8-kube-api-access-8c6pf" (OuterVolumeSpecName: "kube-api-access-8c6pf") pod "1463f956-55d1-4d41-8f37-17cfc2de87c8" (UID: "1463f956-55d1-4d41-8f37-17cfc2de87c8"). InnerVolumeSpecName "kube-api-access-8c6pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:55:37.087878 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.087796 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6pf\" (UniqueName: \"kubernetes.io/projected/1463f956-55d1-4d41-8f37-17cfc2de87c8-kube-api-access-8c6pf\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:55:37.087878 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.087825 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1463f956-55d1-4d41-8f37-17cfc2de87c8-extensions-socket-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 27 23:55:37.510162 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.510136 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" Jul 27 23:55:37.510637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.510136 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl" event={"ID":"1463f956-55d1-4d41-8f37-17cfc2de87c8","Type":"ContainerDied","Data":"8ac6ed72fc5f152fad718ddaf0fcb509126bf32680010ce69be316dfbcaa3e4c"} Jul 27 23:55:37.510637 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.510274 2555 scope.go:117] "RemoveContainer" containerID="58dffd0cf74f8f6552b8e73f35482c19e637018af31afb5b23ae597e69c5fbee" Jul 27 23:55:37.530224 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.530193 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl"] Jul 27 23:55:37.533903 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:37.533878 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zj7kl"] Jul 27 23:55:38.564802 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:55:38.564769 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1463f956-55d1-4d41-8f37-17cfc2de87c8" path="/var/lib/kubelet/pods/1463f956-55d1-4d41-8f37-17cfc2de87c8/volumes" Jul 27 23:56:28.967342 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:56:28.967262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:22.446669 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.446590 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd"] Jul 27 23:59:22.447191 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.447171 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1463f956-55d1-4d41-8f37-17cfc2de87c8" containerName="manager" Jul 27 23:59:22.447242 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.447197 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1463f956-55d1-4d41-8f37-17cfc2de87c8" containerName="manager" Jul 27 23:59:22.447307 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.447295 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1463f956-55d1-4d41-8f37-17cfc2de87c8" containerName="manager" Jul 27 23:59:22.450760 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.450742 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.453210 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.453190 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-aaa4aaee-kserve-self-signed-certs\"" Jul 27 23:59:22.467761 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.467738 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd"] Jul 27 23:59:22.501830 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.501790 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklqr\" (UniqueName: \"kubernetes.io/projected/078f0286-0b76-496c-a6a0-a26c5d3a54f7-kube-api-access-zklqr\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.501996 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.501860 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-model-cache\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.501996 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.501922 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-home\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.501996 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.501990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-dshm\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.502128 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.502049 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tls-certs\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.502128 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.502083 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tmp-dir\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.515640 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.515606 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:22.602486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-model-cache\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.602486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-home\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.602754 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-dshm\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.602754 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602586 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tls-certs\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.602754 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602611 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tmp-dir\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.602754 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602672 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklqr\" (UniqueName: \"kubernetes.io/projected/078f0286-0b76-496c-a6a0-a26c5d3a54f7-kube-api-access-zklqr\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.602950 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602901 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-model-cache\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.603013 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602947 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-home\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.603013 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.602990 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tmp-dir\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.605073 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.605045 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-dshm\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.605073 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.605067 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tls-certs\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.609967 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.609939 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklqr\" (UniqueName: \"kubernetes.io/projected/078f0286-0b76-496c-a6a0-a26c5d3a54f7-kube-api-access-zklqr\") pod \"e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.761542 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.761490 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:22.889486 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:22.889460 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd"] Jul 27 23:59:22.891390 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:59:22.891361 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod078f0286_0b76_496c_a6a0_a26c5d3a54f7.slice/crio-43d332a272a948d63e974576bb937aed0c10e75946156053c0ab366aea44a500 WatchSource:0}: Error finding container 43d332a272a948d63e974576bb937aed0c10e75946156053c0ab366aea44a500: Status 404 returned error can't find the container with id 43d332a272a948d63e974576bb937aed0c10e75946156053c0ab366aea44a500 Jul 27 23:59:23.507860 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:23.507825 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" event={"ID":"078f0286-0b76-496c-a6a0-a26c5d3a54f7","Type":"ContainerStarted","Data":"e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9"} Jul 27 23:59:23.507860 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:23.507868 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" event={"ID":"078f0286-0b76-496c-a6a0-a26c5d3a54f7","Type":"ContainerStarted","Data":"43d332a272a948d63e974576bb937aed0c10e75946156053c0ab366aea44a500"} Jul 27 23:59:23.525905 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:23.525861 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" podStartSLOduration=1.525846383 podStartE2EDuration="1.525846383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:23.52509688 +0000 UTC m=+1363.449305759" watchObservedRunningTime="2026-07-27 23:59:23.525846383 +0000 UTC m=+1363.450055261" Jul 27 23:59:32.762573 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:32.762538 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:32.763055 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:32.762593 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:32.775078 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:32.775053 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:33.562988 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:33.562955 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 27 23:59:38.028309 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:38.028276 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:49.140995 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.140961 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x"] Jul 27 23:59:49.144672 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.144650 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.146934 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.146916 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-aaa4aaee-kserve-self-signed-certs\"" Jul 27 23:59:49.154440 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.154416 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x"] Jul 27 23:59:49.181169 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.181142 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:49.234807 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.234781 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-home\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.234924 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.234809 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-dshm\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.234924 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.234833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-model-cache\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.235012 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.234937 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tls-certs\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.235012 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.234978 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lx9\" (UniqueName: \"kubernetes.io/projected/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-kube-api-access-k8lx9\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.235085 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.235047 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tmp-dir\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336216 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tls-certs\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336374 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336229 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lx9\" (UniqueName: \"kubernetes.io/projected/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-kube-api-access-k8lx9\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336374 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336263 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tmp-dir\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336374 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336293 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-home\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336374 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336307 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-dshm\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336374 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336323 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-model-cache\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336775 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336746 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tmp-dir\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336775 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-model-cache\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.336854 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.336807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-home\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.338554 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.338530 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-dshm\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.338732 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.338715 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tls-certs\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.343976 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.343953 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lx9\" (UniqueName: \"kubernetes.io/projected/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-kube-api-access-k8lx9\") pod \"e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.457160 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.457068 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:49.590022 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.589998 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x"] Jul 27 23:59:49.591062 ip-10-0-130-94 kubenswrapper[2555]: W0727 23:59:49.591023 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode872d270_c9ff_4c86_a2fe_dbf7b81e08d6.slice/crio-83d4987e0ec1779bb1f4244295a46afddb86773322899c78a4f3d32ce61aade6 WatchSource:0}: Error finding container 83d4987e0ec1779bb1f4244295a46afddb86773322899c78a4f3d32ce61aade6: Status 404 returned error can't find the container with id 83d4987e0ec1779bb1f4244295a46afddb86773322899c78a4f3d32ce61aade6 Jul 27 23:59:49.621870 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:49.621843 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" event={"ID":"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6","Type":"ContainerStarted","Data":"83d4987e0ec1779bb1f4244295a46afddb86773322899c78a4f3d32ce61aade6"} Jul 27 23:59:50.627837 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:50.627805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" event={"ID":"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6","Type":"ContainerStarted","Data":"7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2"} Jul 27 23:59:50.644383 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:50.644332 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" podStartSLOduration=1.644317365 podStartE2EDuration="1.644317365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:50.643048713 +0000 UTC m=+1390.567257592" watchObservedRunningTime="2026-07-27 23:59:50.644317365 +0000 UTC m=+1390.568526243" Jul 27 23:59:59.457384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:59.457334 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:59.457384 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:59.457397 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:59.470039 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:59.470012 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 27 23:59:59.678813 ip-10-0-130-94 kubenswrapper[2555]: I0727 23:59:59.678786 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 28 00:00:00.140610 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.140575 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29753280-bgs47"] Jul 28 00:00:00.144218 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.144201 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.146836 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.146814 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-888np\"" Jul 28 00:00:00.146948 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.146814 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 28 00:00:00.153365 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.153335 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29753280-bgs47"] Jul 28 00:00:00.334221 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.334181 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60da7601-7076-476c-a357-1cc8f6a28fe4-serviceca\") pod \"image-pruner-29753280-bgs47\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.334387 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.334302 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptkq\" (UniqueName: \"kubernetes.io/projected/60da7601-7076-476c-a357-1cc8f6a28fe4-kube-api-access-pptkq\") pod \"image-pruner-29753280-bgs47\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.435127 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.435037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptkq\" (UniqueName: \"kubernetes.io/projected/60da7601-7076-476c-a357-1cc8f6a28fe4-kube-api-access-pptkq\") pod \"image-pruner-29753280-bgs47\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.435261 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.435169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60da7601-7076-476c-a357-1cc8f6a28fe4-serviceca\") pod \"image-pruner-29753280-bgs47\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.435810 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.435788 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60da7601-7076-476c-a357-1cc8f6a28fe4-serviceca\") pod \"image-pruner-29753280-bgs47\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.443964 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.443937 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptkq\" (UniqueName: \"kubernetes.io/projected/60da7601-7076-476c-a357-1cc8f6a28fe4-kube-api-access-pptkq\") pod \"image-pruner-29753280-bgs47\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.471540 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.471487 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:00.612808 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.612779 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29753280-bgs47"] Jul 28 00:00:00.614837 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:00:00.614795 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60da7601_7076_476c_a357_1cc8f6a28fe4.slice/crio-6a546bda08bf888a1bff39616fdfeeeaf2db5fed233c3e572788323502edd9e9 WatchSource:0}: Error finding container 6a546bda08bf888a1bff39616fdfeeeaf2db5fed233c3e572788323502edd9e9: Status 404 returned error can't find the container with id 6a546bda08bf888a1bff39616fdfeeeaf2db5fed233c3e572788323502edd9e9 Jul 28 00:00:00.672810 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:00.672774 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-bgs47" event={"ID":"60da7601-7076-476c-a357-1cc8f6a28fe4","Type":"ContainerStarted","Data":"6a546bda08bf888a1bff39616fdfeeeaf2db5fed233c3e572788323502edd9e9"} Jul 28 00:00:01.679045 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:01.679010 2555 generic.go:356] "Generic (PLEG): container finished" podID="60da7601-7076-476c-a357-1cc8f6a28fe4" containerID="a33740ad2bd6a48de52e9959cd275f0d900109f0c1b9c50fccfa380f0e605cc9" exitCode=0 Jul 28 00:00:01.679426 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:01.679092 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-bgs47" event={"ID":"60da7601-7076-476c-a357-1cc8f6a28fe4","Type":"ContainerDied","Data":"a33740ad2bd6a48de52e9959cd275f0d900109f0c1b9c50fccfa380f0e605cc9"} Jul 28 00:00:02.815017 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:02.814996 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:02.957023 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:02.956921 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pptkq\" (UniqueName: \"kubernetes.io/projected/60da7601-7076-476c-a357-1cc8f6a28fe4-kube-api-access-pptkq\") pod \"60da7601-7076-476c-a357-1cc8f6a28fe4\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " Jul 28 00:00:02.957023 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:02.956973 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60da7601-7076-476c-a357-1cc8f6a28fe4-serviceca\") pod \"60da7601-7076-476c-a357-1cc8f6a28fe4\" (UID: \"60da7601-7076-476c-a357-1cc8f6a28fe4\") " Jul 28 00:00:02.957348 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:02.957323 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60da7601-7076-476c-a357-1cc8f6a28fe4-serviceca" (OuterVolumeSpecName: "serviceca") pod "60da7601-7076-476c-a357-1cc8f6a28fe4" (UID: "60da7601-7076-476c-a357-1cc8f6a28fe4"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:00:02.959066 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:02.959042 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60da7601-7076-476c-a357-1cc8f6a28fe4-kube-api-access-pptkq" (OuterVolumeSpecName: "kube-api-access-pptkq") pod "60da7601-7076-476c-a357-1cc8f6a28fe4" (UID: "60da7601-7076-476c-a357-1cc8f6a28fe4"). InnerVolumeSpecName "kube-api-access-pptkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:00:03.057915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:03.057862 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptkq\" (UniqueName: \"kubernetes.io/projected/60da7601-7076-476c-a357-1cc8f6a28fe4-kube-api-access-pptkq\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:03.057915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:03.057907 2555 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60da7601-7076-476c-a357-1cc8f6a28fe4-serviceca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:03.689631 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:03.689605 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-bgs47" Jul 28 00:00:03.689809 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:03.689595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-bgs47" event={"ID":"60da7601-7076-476c-a357-1cc8f6a28fe4","Type":"ContainerDied","Data":"6a546bda08bf888a1bff39616fdfeeeaf2db5fed233c3e572788323502edd9e9"} Jul 28 00:00:03.689809 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:03.689732 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a546bda08bf888a1bff39616fdfeeeaf2db5fed233c3e572788323502edd9e9" Jul 28 00:00:21.058678 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:21.058646 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd"] Jul 28 00:00:21.089114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:21.089083 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:00:22.277550 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:22.277509 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x"] Jul 28 00:00:22.282714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:22.282649 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:00:36.078089 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.078025 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" podUID="078f0286-0b76-496c-a6a0-a26c5d3a54f7" containerName="main" containerID="cri-o://e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9" gracePeriod=45 Jul 28 00:00:36.329514 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.329428 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 28 00:00:36.453984 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.453951 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tls-certs\") pod \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " Jul 28 00:00:36.454186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454015 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-home\") pod \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " Jul 28 00:00:36.454186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454148 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zklqr\" (UniqueName: \"kubernetes.io/projected/078f0286-0b76-496c-a6a0-a26c5d3a54f7-kube-api-access-zklqr\") pod \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " Jul 28 00:00:36.454308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454188 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tmp-dir\") pod \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " Jul 28 00:00:36.454308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454246 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-model-cache\") pod \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " Jul 28 00:00:36.454308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454272 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-dshm\") pod \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\" (UID: \"078f0286-0b76-496c-a6a0-a26c5d3a54f7\") " Jul 28 00:00:36.454308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454283 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-home" (OuterVolumeSpecName: "home") pod "078f0286-0b76-496c-a6a0-a26c5d3a54f7" (UID: "078f0286-0b76-496c-a6a0-a26c5d3a54f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:36.454601 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454582 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:36.454762 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454731 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "078f0286-0b76-496c-a6a0-a26c5d3a54f7" (UID: "078f0286-0b76-496c-a6a0-a26c5d3a54f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:36.454828 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.454775 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "078f0286-0b76-496c-a6a0-a26c5d3a54f7" (UID: "078f0286-0b76-496c-a6a0-a26c5d3a54f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:36.456465 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.456436 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "078f0286-0b76-496c-a6a0-a26c5d3a54f7" (UID: "078f0286-0b76-496c-a6a0-a26c5d3a54f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:00:36.456578 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.456482 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078f0286-0b76-496c-a6a0-a26c5d3a54f7-kube-api-access-zklqr" (OuterVolumeSpecName: "kube-api-access-zklqr") pod "078f0286-0b76-496c-a6a0-a26c5d3a54f7" (UID: "078f0286-0b76-496c-a6a0-a26c5d3a54f7"). InnerVolumeSpecName "kube-api-access-zklqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:00:36.456754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.456734 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-dshm" (OuterVolumeSpecName: "dshm") pod "078f0286-0b76-496c-a6a0-a26c5d3a54f7" (UID: "078f0286-0b76-496c-a6a0-a26c5d3a54f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:36.555969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.555933 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklqr\" (UniqueName: \"kubernetes.io/projected/078f0286-0b76-496c-a6a0-a26c5d3a54f7-kube-api-access-zklqr\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:36.555969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.555965 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:36.556180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.555976 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:36.556180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.555987 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f0286-0b76-496c-a6a0-a26c5d3a54f7-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:36.556180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.555996 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f0286-0b76-496c-a6a0-a26c5d3a54f7-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:36.853464 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.853431 2555 generic.go:356] "Generic (PLEG): container finished" podID="078f0286-0b76-496c-a6a0-a26c5d3a54f7" containerID="e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9" exitCode=0 Jul 28 00:00:36.853667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.853476 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" Jul 28 00:00:36.853667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.853517 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" event={"ID":"078f0286-0b76-496c-a6a0-a26c5d3a54f7","Type":"ContainerDied","Data":"e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9"} Jul 28 00:00:36.853667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.853567 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd" event={"ID":"078f0286-0b76-496c-a6a0-a26c5d3a54f7","Type":"ContainerDied","Data":"43d332a272a948d63e974576bb937aed0c10e75946156053c0ab366aea44a500"} Jul 28 00:00:36.853667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.853585 2555 scope.go:117] "RemoveContainer" containerID="e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9" Jul 28 00:00:36.862584 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.862568 2555 scope.go:117] "RemoveContainer" containerID="e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9" Jul 28 00:00:36.862880 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:00:36.862851 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9\": container with ID starting with e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9 not found: ID does not exist" containerID="e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9" Jul 28 00:00:36.862971 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.862885 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9"} err="failed to get container status \"e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9\": rpc error: code = NotFound desc = could not find container \"e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9\": container with ID starting with e49eacd82b74d1d04e0847085faeb3f36d8810f2a8fbc76f6dc12949356086d9 not found: ID does not exist" Jul 28 00:00:36.868269 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.868246 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd"] Jul 28 00:00:36.873794 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:36.873774 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-aaa4aaee-kserve-6d9c9dd4f9-nfvpd"] Jul 28 00:00:37.297680 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.297642 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" podUID="e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" containerName="main" containerID="cri-o://7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2" gracePeriod=45 Jul 28 00:00:37.540385 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.540360 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 28 00:00:37.668313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668212 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-home\") pod \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " Jul 28 00:00:37.668313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668276 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tmp-dir\") pod \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " Jul 28 00:00:37.668596 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668332 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-model-cache\") pod \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " Jul 28 00:00:37.668596 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668356 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8lx9\" (UniqueName: \"kubernetes.io/projected/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-kube-api-access-k8lx9\") pod \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " Jul 28 00:00:37.668596 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668383 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-dshm\") pod \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " Jul 28 00:00:37.668596 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668415 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tls-certs\") pod \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\" (UID: \"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6\") " Jul 28 00:00:37.668818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668604 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-home" (OuterVolumeSpecName: "home") pod "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" (UID: "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:37.668818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668644 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" (UID: "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:37.668818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668661 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" (UID: "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:37.668818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668710 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:37.668818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.668727 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:37.670569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.670538 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-dshm" (OuterVolumeSpecName: "dshm") pod "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" (UID: "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:00:37.670569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.670543 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" (UID: "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:00:37.670705 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.670633 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-kube-api-access-k8lx9" (OuterVolumeSpecName: "kube-api-access-k8lx9") pod "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" (UID: "e872d270-c9ff-4c86-a2fe-dbf7b81e08d6"). InnerVolumeSpecName "kube-api-access-k8lx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:00:37.769723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.769691 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8lx9\" (UniqueName: \"kubernetes.io/projected/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-kube-api-access-k8lx9\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:37.769723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.769719 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:37.769723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.769729 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:37.769980 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.769739 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:00:37.858891 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.858858 2555 generic.go:356] "Generic (PLEG): container finished" podID="e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" containerID="7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2" exitCode=0 Jul 28 00:00:37.859113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.858898 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" Jul 28 00:00:37.859113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.858953 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" event={"ID":"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6","Type":"ContainerDied","Data":"7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2"} Jul 28 00:00:37.859113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.858999 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x" event={"ID":"e872d270-c9ff-4c86-a2fe-dbf7b81e08d6","Type":"ContainerDied","Data":"83d4987e0ec1779bb1f4244295a46afddb86773322899c78a4f3d32ce61aade6"} Jul 28 00:00:37.859113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.859022 2555 scope.go:117] "RemoveContainer" containerID="7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2" Jul 28 00:00:37.868308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.868288 2555 scope.go:117] "RemoveContainer" containerID="7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2" Jul 28 00:00:37.868584 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:00:37.868567 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2\": container with ID starting with 7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2 not found: ID does not exist" containerID="7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2" Jul 28 00:00:37.868646 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.868591 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2"} err="failed to get container status \"7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2\": rpc error: code = NotFound desc = could not find container \"7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2\": container with ID starting with 7ca6225f9f1a823cac93ec06a35d97fc294a13643a486d48edaf82a62fbbb3c2 not found: ID does not exist" Jul 28 00:00:37.879928 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.879897 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x"] Jul 28 00:00:37.881562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:37.881545 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-aaa4aaee-kserve-566d7b6f79-dvl7x"] Jul 28 00:00:38.565277 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:38.565232 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078f0286-0b76-496c-a6a0-a26c5d3a54f7" path="/var/lib/kubelet/pods/078f0286-0b76-496c-a6a0-a26c5d3a54f7/volumes" Jul 28 00:00:38.565853 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:00:38.565831 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" path="/var/lib/kubelet/pods/e872d270-c9ff-4c86-a2fe-dbf7b81e08d6/volumes" Jul 28 00:01:20.400162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:01:20.400129 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:03:39.560603 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:03:39.560508 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:01.135893 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:01.135859 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:04.176627 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:04.176587 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:04.274168 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:04.274131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:04.653791 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:04.653756 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:10.323395 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:10.323349 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:10.447043 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:10.447009 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:10.544154 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:10.544117 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:10.834395 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:10.834358 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:11.127949 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:11.127803 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:13.627544 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.627508 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:13.714843 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.714801 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:13.715122 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715102 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k"] Jul 28 00:04:13.715522 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715486 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60da7601-7076-476c-a357-1cc8f6a28fe4" containerName="image-pruner" Jul 28 00:04:13.715522 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715516 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60da7601-7076-476c-a357-1cc8f6a28fe4" containerName="image-pruner" Jul 28 00:04:13.715607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715540 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" containerName="main" Jul 28 00:04:13.715607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715548 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" containerName="main" Jul 28 00:04:13.715607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715566 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078f0286-0b76-496c-a6a0-a26c5d3a54f7" containerName="main" Jul 28 00:04:13.715607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715574 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="078f0286-0b76-496c-a6a0-a26c5d3a54f7" containerName="main" Jul 28 00:04:13.715733 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715640 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e872d270-c9ff-4c86-a2fe-dbf7b81e08d6" containerName="main" Jul 28 00:04:13.715733 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715650 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="60da7601-7076-476c-a357-1cc8f6a28fe4" containerName="image-pruner" Jul 28 00:04:13.715733 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.715663 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="078f0286-0b76-496c-a6a0-a26c5d3a54f7" containerName="main" Jul 28 00:04:13.718515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.718481 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.720807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.720781 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-aa509c8d-serving-cert\"" Jul 28 00:04:13.728936 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.728915 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k"] Jul 28 00:04:13.790221 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.790182 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:13.840923 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.840892 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/970a0b28-f5f5-47f0-973f-bd5f176333b3-kube-api-access-vwbv2\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.841107 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.840945 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970a0b28-f5f5-47f0-973f-bd5f176333b3-openshift-service-ca\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.841107 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.841010 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970a0b28-f5f5-47f0-973f-bd5f176333b3-maas-api-tls\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.895780 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.895674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:13.941976 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.941941 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970a0b28-f5f5-47f0-973f-bd5f176333b3-maas-api-tls\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.942143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.941993 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/970a0b28-f5f5-47f0-973f-bd5f176333b3-kube-api-access-vwbv2\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.942143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.942032 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970a0b28-f5f5-47f0-973f-bd5f176333b3-openshift-service-ca\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.942694 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.942675 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970a0b28-f5f5-47f0-973f-bd5f176333b3-openshift-service-ca\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.944341 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.944308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970a0b28-f5f5-47f0-973f-bd5f176333b3-maas-api-tls\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:13.949849 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:13.949827 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/970a0b28-f5f5-47f0-973f-bd5f176333b3-kube-api-access-vwbv2\") pod \"maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:14.031644 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.031614 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:14.170564 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.170541 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k"] Jul 28 00:04:14.172739 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:14.172710 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970a0b28_f5f5_47f0_973f_bd5f176333b3.slice/crio-bae5de8278a17ed3040b39c74802b1b991272a57205f0d67750ef4678dd38835 WatchSource:0}: Error finding container bae5de8278a17ed3040b39c74802b1b991272a57205f0d67750ef4678dd38835: Status 404 returned error can't find the container with id bae5de8278a17ed3040b39c74802b1b991272a57205f0d67750ef4678dd38835 Jul 28 00:04:14.174245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.174225 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:04:14.317838 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.317808 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:14.404844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.404806 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:14.409877 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.409852 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl"] Jul 28 00:04:14.414362 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.414345 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.416212 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.416192 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aa509c8d\"" Jul 28 00:04:14.416294 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.416276 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aa509c8d-dockercfg-gh58l\"" Jul 28 00:04:14.421562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.421507 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl"] Jul 28 00:04:14.450804 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.450777 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5"] Jul 28 00:04:14.454014 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.453999 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.466311 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.466290 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5"] Jul 28 00:04:14.548273 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.548241 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wljf\" (UniqueName: \"kubernetes.io/projected/92f8e69e-c733-4fcc-b39e-4b2b429d8462-kube-api-access-9wljf\") pod \"payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.548458 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.548295 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f8e69e-c733-4fcc-b39e-4b2b429d8462-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.548458 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.548318 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa44049d-03b4-4097-a203-dfd267b28175-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.548458 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.548353 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmpd\" (UniqueName: \"kubernetes.io/projected/aa44049d-03b4-4097-a203-dfd267b28175-kube-api-access-9xmpd\") pod \"payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.649812 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.649772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wljf\" (UniqueName: \"kubernetes.io/projected/92f8e69e-c733-4fcc-b39e-4b2b429d8462-kube-api-access-9wljf\") pod \"payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.649812 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.649812 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f8e69e-c733-4fcc-b39e-4b2b429d8462-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.650285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.649831 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa44049d-03b4-4097-a203-dfd267b28175-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.650285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.649900 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmpd\" (UniqueName: \"kubernetes.io/projected/aa44049d-03b4-4097-a203-dfd267b28175-kube-api-access-9xmpd\") pod \"payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.650521 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.650481 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa44049d-03b4-4097-a203-dfd267b28175-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.650678 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.650662 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f8e69e-c733-4fcc-b39e-4b2b429d8462-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.658457 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.658429 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmpd\" (UniqueName: \"kubernetes.io/projected/aa44049d-03b4-4097-a203-dfd267b28175-kube-api-access-9xmpd\") pod \"payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.659138 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.659117 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wljf\" (UniqueName: \"kubernetes.io/projected/92f8e69e-c733-4fcc-b39e-4b2b429d8462-kube-api-access-9wljf\") pod \"payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.726215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.726136 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:14.764371 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.764339 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:14.845535 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.844810 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" event={"ID":"970a0b28-f5f5-47f0-973f-bd5f176333b3","Type":"ContainerStarted","Data":"bae5de8278a17ed3040b39c74802b1b991272a57205f0d67750ef4678dd38835"} Jul 28 00:04:14.913427 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:14.913402 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5"] Jul 28 00:04:14.916667 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:14.916635 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f8e69e_c733_4fcc_b39e_4b2b429d8462.slice/crio-7e796e55cbac2668770957ca840e731a5b5d4d7e50d52a32871b3d2a313a4145 WatchSource:0}: Error finding container 7e796e55cbac2668770957ca840e731a5b5d4d7e50d52a32871b3d2a313a4145: Status 404 returned error can't find the container with id 7e796e55cbac2668770957ca840e731a5b5d4d7e50d52a32871b3d2a313a4145 Jul 28 00:04:15.099627 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.099595 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl"] Jul 28 00:04:15.101639 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:15.101613 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa44049d_03b4_4097_a203_dfd267b28175.slice/crio-787d391892de1e047b4a5c4ac18cb1be177eca2d5a900c5d600ce396010e1304 WatchSource:0}: Error finding container 787d391892de1e047b4a5c4ac18cb1be177eca2d5a900c5d600ce396010e1304: Status 404 returned error can't find the container with id 787d391892de1e047b4a5c4ac18cb1be177eca2d5a900c5d600ce396010e1304 Jul 28 00:04:15.851204 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.851164 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" event={"ID":"970a0b28-f5f5-47f0-973f-bd5f176333b3","Type":"ContainerStarted","Data":"6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731"} Jul 28 00:04:15.851779 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.851245 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:15.852717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.852691 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" event={"ID":"aa44049d-03b4-4097-a203-dfd267b28175","Type":"ContainerStarted","Data":"cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8"} Jul 28 00:04:15.852849 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.852736 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" event={"ID":"aa44049d-03b4-4097-a203-dfd267b28175","Type":"ContainerStarted","Data":"787d391892de1e047b4a5c4ac18cb1be177eca2d5a900c5d600ce396010e1304"} Jul 28 00:04:15.852849 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.852782 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:15.854162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.854145 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" event={"ID":"92f8e69e-c733-4fcc-b39e-4b2b429d8462","Type":"ContainerStarted","Data":"f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc"} Jul 28 00:04:15.854259 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.854174 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" event={"ID":"92f8e69e-c733-4fcc-b39e-4b2b429d8462","Type":"ContainerStarted","Data":"7e796e55cbac2668770957ca840e731a5b5d4d7e50d52a32871b3d2a313a4145"} Jul 28 00:04:15.854259 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.854193 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:15.868971 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.868930 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" podStartSLOduration=2.178487838 podStartE2EDuration="2.8689171s" totalImagesPullingTime="690.429262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:04:15.867089867 +0000 UTC m=+1655.791298745" watchObservedRunningTime="2026-07-28 00:04:15.8689171 +0000 UTC m=+1655.793125977" Jul 28 00:04:15.880557 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.880515 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" podStartSLOduration=1.880485006 podStartE2EDuration="1.880485006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:04:15.879919365 +0000 UTC m=+1655.804128267" watchObservedRunningTime="2026-07-28 00:04:15.880485006 +0000 UTC m=+1655.804693889" Jul 28 00:04:15.894554 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:15.894505 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" podStartSLOduration=1.894479439 podStartE2EDuration="1.894479439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:04:15.893596654 +0000 UTC m=+1655.817805533" watchObservedRunningTime="2026-07-28 00:04:15.894479439 +0000 UTC m=+1655.818688317" Jul 28 00:04:17.477232 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:17.477194 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:21.865279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:21.865244 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:26.862363 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:26.862327 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:26.862865 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:26.862634 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:39.226361 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.226321 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47"] Jul 28 00:04:39.231054 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.231035 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:39.234023 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.234003 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:04:39.262891 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.262860 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47"] Jul 28 00:04:39.368279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.368243 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd71f42b-5d36-4308-ae8c-f80b4277350e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47\" (UID: \"dd71f42b-5d36-4308-ae8c-f80b4277350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:39.469087 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.469053 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd71f42b-5d36-4308-ae8c-f80b4277350e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47\" (UID: \"dd71f42b-5d36-4308-ae8c-f80b4277350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:39.469734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.469712 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd71f42b-5d36-4308-ae8c-f80b4277350e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47\" (UID: \"dd71f42b-5d36-4308-ae8c-f80b4277350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:39.541643 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.541610 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:39.673904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.673866 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47"] Jul 28 00:04:39.674596 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:39.674568 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd71f42b_5d36_4308_ae8c_f80b4277350e.slice/crio-a7a09ab0145958d5ead8888f4a55a5a8e7881e64002b09d5ecd023b0eb08110b WatchSource:0}: Error finding container a7a09ab0145958d5ead8888f4a55a5a8e7881e64002b09d5ecd023b0eb08110b: Status 404 returned error can't find the container with id a7a09ab0145958d5ead8888f4a55a5a8e7881e64002b09d5ecd023b0eb08110b Jul 28 00:04:39.971865 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:39.971779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" event={"ID":"dd71f42b-5d36-4308-ae8c-f80b4277350e","Type":"ContainerStarted","Data":"a7a09ab0145958d5ead8888f4a55a5a8e7881e64002b09d5ecd023b0eb08110b"} Jul 28 00:04:42.989083 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:42.988999 2555 generic.go:356] "Generic (PLEG): container finished" podID="dd71f42b-5d36-4308-ae8c-f80b4277350e" containerID="7f04446f3fcd3e9e49f179dda9b58ab7b6f6a68d8b6ee322a2aeb78c81eee996" exitCode=0 Jul 28 00:04:42.989442 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:42.989083 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" event={"ID":"dd71f42b-5d36-4308-ae8c-f80b4277350e","Type":"ContainerDied","Data":"7f04446f3fcd3e9e49f179dda9b58ab7b6f6a68d8b6ee322a2aeb78c81eee996"} Jul 28 00:04:44.134565 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.134539 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:44.211767 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.211734 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd71f42b-5d36-4308-ae8c-f80b4277350e-maas-api-service-ca\") pod \"dd71f42b-5d36-4308-ae8c-f80b4277350e\" (UID: \"dd71f42b-5d36-4308-ae8c-f80b4277350e\") " Jul 28 00:04:44.212117 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.212095 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd71f42b-5d36-4308-ae8c-f80b4277350e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd71f42b-5d36-4308-ae8c-f80b4277350e" (UID: "dd71f42b-5d36-4308-ae8c-f80b4277350e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:04:44.312359 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.312270 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd71f42b-5d36-4308-ae8c-f80b4277350e-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:44.999464 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.999436 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" Jul 28 00:04:44.999678 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.999473 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47" event={"ID":"dd71f42b-5d36-4308-ae8c-f80b4277350e","Type":"ContainerDied","Data":"a7a09ab0145958d5ead8888f4a55a5a8e7881e64002b09d5ecd023b0eb08110b"} Jul 28 00:04:44.999678 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:44.999551 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a09ab0145958d5ead8888f4a55a5a8e7881e64002b09d5ecd023b0eb08110b" Jul 28 00:04:49.584718 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.584676 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k"] Jul 28 00:04:49.585229 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.584927 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" podUID="970a0b28-f5f5-47f0-973f-bd5f176333b3" containerName="maas-api" containerID="cri-o://6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731" gracePeriod=30 Jul 28 00:04:49.606925 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.606894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:49.628837 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.628805 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5"] Jul 28 00:04:49.629045 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.629017 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" podUID="92f8e69e-c733-4fcc-b39e-4b2b429d8462" containerName="payload-processing" containerID="cri-o://f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc" gracePeriod=30 Jul 28 00:04:49.637163 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.637130 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl"] Jul 28 00:04:49.637683 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.637625 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" podUID="aa44049d-03b4-4097-a203-dfd267b28175" containerName="payload-pre-processing" containerID="cri-o://cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8" gracePeriod=30 Jul 28 00:04:49.918027 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.917999 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:49.920412 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.920382 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:49.949551 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.949521 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:49.960143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960120 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/970a0b28-f5f5-47f0-973f-bd5f176333b3-kube-api-access-vwbv2\") pod \"970a0b28-f5f5-47f0-973f-bd5f176333b3\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " Jul 28 00:04:49.960311 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960297 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970a0b28-f5f5-47f0-973f-bd5f176333b3-maas-api-tls\") pod \"970a0b28-f5f5-47f0-973f-bd5f176333b3\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " Jul 28 00:04:49.960395 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960327 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f8e69e-c733-4fcc-b39e-4b2b429d8462-plugins-config-volume\") pod \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " Jul 28 00:04:49.960395 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960372 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970a0b28-f5f5-47f0-973f-bd5f176333b3-openshift-service-ca\") pod \"970a0b28-f5f5-47f0-973f-bd5f176333b3\" (UID: \"970a0b28-f5f5-47f0-973f-bd5f176333b3\") " Jul 28 00:04:49.960528 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960416 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wljf\" (UniqueName: \"kubernetes.io/projected/92f8e69e-c733-4fcc-b39e-4b2b429d8462-kube-api-access-9wljf\") pod \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\" (UID: \"92f8e69e-c733-4fcc-b39e-4b2b429d8462\") " Jul 28 00:04:49.960768 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960742 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f8e69e-c733-4fcc-b39e-4b2b429d8462-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92f8e69e-c733-4fcc-b39e-4b2b429d8462" (UID: "92f8e69e-c733-4fcc-b39e-4b2b429d8462"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:04:49.960860 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.960765 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970a0b28-f5f5-47f0-973f-bd5f176333b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "970a0b28-f5f5-47f0-973f-bd5f176333b3" (UID: "970a0b28-f5f5-47f0-973f-bd5f176333b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:04:49.962777 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.962749 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f8e69e-c733-4fcc-b39e-4b2b429d8462-kube-api-access-9wljf" (OuterVolumeSpecName: "kube-api-access-9wljf") pod "92f8e69e-c733-4fcc-b39e-4b2b429d8462" (UID: "92f8e69e-c733-4fcc-b39e-4b2b429d8462"). InnerVolumeSpecName "kube-api-access-9wljf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:04:49.962931 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.962825 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970a0b28-f5f5-47f0-973f-bd5f176333b3-kube-api-access-vwbv2" (OuterVolumeSpecName: "kube-api-access-vwbv2") pod "970a0b28-f5f5-47f0-973f-bd5f176333b3" (UID: "970a0b28-f5f5-47f0-973f-bd5f176333b3"). InnerVolumeSpecName "kube-api-access-vwbv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:04:49.962931 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:49.962844 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970a0b28-f5f5-47f0-973f-bd5f176333b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "970a0b28-f5f5-47f0-973f-bd5f176333b3" (UID: "970a0b28-f5f5-47f0-973f-bd5f176333b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:04:50.023721 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.023686 2555 generic.go:356] "Generic (PLEG): container finished" podID="92f8e69e-c733-4fcc-b39e-4b2b429d8462" containerID="f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc" exitCode=0 Jul 28 00:04:50.023915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.023732 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" Jul 28 00:04:50.023915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.023778 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" event={"ID":"92f8e69e-c733-4fcc-b39e-4b2b429d8462","Type":"ContainerDied","Data":"f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc"} Jul 28 00:04:50.023915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.023832 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5" event={"ID":"92f8e69e-c733-4fcc-b39e-4b2b429d8462","Type":"ContainerDied","Data":"7e796e55cbac2668770957ca840e731a5b5d4d7e50d52a32871b3d2a313a4145"} Jul 28 00:04:50.023915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.023858 2555 scope.go:117] "RemoveContainer" containerID="f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc" Jul 28 00:04:50.025077 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.025053 2555 generic.go:356] "Generic (PLEG): container finished" podID="970a0b28-f5f5-47f0-973f-bd5f176333b3" containerID="6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731" exitCode=0 Jul 28 00:04:50.025180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.025084 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" event={"ID":"970a0b28-f5f5-47f0-973f-bd5f176333b3","Type":"ContainerDied","Data":"6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731"} Jul 28 00:04:50.025180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.025117 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" Jul 28 00:04:50.025295 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.025126 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k" event={"ID":"970a0b28-f5f5-47f0-973f-bd5f176333b3","Type":"ContainerDied","Data":"bae5de8278a17ed3040b39c74802b1b991272a57205f0d67750ef4678dd38835"} Jul 28 00:04:50.026481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.026461 2555 generic.go:356] "Generic (PLEG): container finished" podID="aa44049d-03b4-4097-a203-dfd267b28175" containerID="cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8" exitCode=0 Jul 28 00:04:50.026598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.026531 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" Jul 28 00:04:50.026598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.026559 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" event={"ID":"aa44049d-03b4-4097-a203-dfd267b28175","Type":"ContainerDied","Data":"cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8"} Jul 28 00:04:50.026686 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.026613 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl" event={"ID":"aa44049d-03b4-4097-a203-dfd267b28175","Type":"ContainerDied","Data":"787d391892de1e047b4a5c4ac18cb1be177eca2d5a900c5d600ce396010e1304"} Jul 28 00:04:50.033266 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.033249 2555 scope.go:117] "RemoveContainer" containerID="f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc" Jul 28 00:04:50.033563 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:04:50.033539 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc\": container with ID starting with f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc not found: ID does not exist" containerID="f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc" Jul 28 00:04:50.033667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.033569 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc"} err="failed to get container status \"f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc\": rpc error: code = NotFound desc = could not find container \"f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc\": container with ID starting with f8cec8dc50176308d5a5a7ea37b74f292d76391bc30888f85833f77965ad14dc not found: ID does not exist" Jul 28 00:04:50.033667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.033592 2555 scope.go:117] "RemoveContainer" containerID="6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731" Jul 28 00:04:50.041675 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.041661 2555 scope.go:117] "RemoveContainer" containerID="6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731" Jul 28 00:04:50.041914 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:04:50.041897 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731\": container with ID starting with 6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731 not found: ID does not exist" containerID="6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731" Jul 28 00:04:50.041954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.041921 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731"} err="failed to get container status \"6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731\": rpc error: code = NotFound desc = could not find container \"6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731\": container with ID starting with 6cf26e6a8c2edd5deaa8442c10ea1bd1b1a09788ed648b4a00e6e147e6ceb731 not found: ID does not exist" Jul 28 00:04:50.041954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.041935 2555 scope.go:117] "RemoveContainer" containerID="cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8" Jul 28 00:04:50.048703 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.048678 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k"] Jul 28 00:04:50.049978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.049958 2555 scope.go:117] "RemoveContainer" containerID="cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8" Jul 28 00:04:50.050244 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:04:50.050223 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8\": container with ID starting with cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8 not found: ID does not exist" containerID="cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8" Jul 28 00:04:50.050346 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.050249 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8"} err="failed to get container status \"cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8\": rpc error: code = NotFound desc = could not find container \"cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8\": container with ID starting with cb4339f9cbcb8d89b9bd0cfe5fad9be4c74a7dad2d9efac6debb73cb6dce1cb8 not found: ID does not exist" Jul 28 00:04:50.050557 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.050478 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa509c8d-758fcc7785-f7x9k"] Jul 28 00:04:50.058972 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.058953 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5"] Jul 28 00:04:50.061261 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061219 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa44049d-03b4-4097-a203-dfd267b28175-plugins-config-volume\") pod \"aa44049d-03b4-4097-a203-dfd267b28175\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " Jul 28 00:04:50.061586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061284 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xmpd\" (UniqueName: \"kubernetes.io/projected/aa44049d-03b4-4097-a203-dfd267b28175-kube-api-access-9xmpd\") pod \"aa44049d-03b4-4097-a203-dfd267b28175\" (UID: \"aa44049d-03b4-4097-a203-dfd267b28175\") " Jul 28 00:04:50.061586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061550 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970a0b28-f5f5-47f0-973f-bd5f176333b3-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.061586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061572 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f8e69e-c733-4fcc-b39e-4b2b429d8462-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.061586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061586 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970a0b28-f5f5-47f0-973f-bd5f176333b3-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.061840 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061599 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wljf\" (UniqueName: \"kubernetes.io/projected/92f8e69e-c733-4fcc-b39e-4b2b429d8462-kube-api-access-9wljf\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.061840 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061612 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/970a0b28-f5f5-47f0-973f-bd5f176333b3-kube-api-access-vwbv2\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.061933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.061841 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa44049d-03b4-4097-a203-dfd267b28175-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa44049d-03b4-4097-a203-dfd267b28175" (UID: "aa44049d-03b4-4097-a203-dfd267b28175"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:04:50.062927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.062910 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa509c8d-576d66dd5d-f7jp5"] Jul 28 00:04:50.063471 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.063452 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa44049d-03b4-4097-a203-dfd267b28175-kube-api-access-9xmpd" (OuterVolumeSpecName: "kube-api-access-9xmpd") pod "aa44049d-03b4-4097-a203-dfd267b28175" (UID: "aa44049d-03b4-4097-a203-dfd267b28175"). InnerVolumeSpecName "kube-api-access-9xmpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:04:50.167703 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.162338 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa44049d-03b4-4097-a203-dfd267b28175-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.167703 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.162378 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xmpd\" (UniqueName: \"kubernetes.io/projected/aa44049d-03b4-4097-a203-dfd267b28175-kube-api-access-9xmpd\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:04:50.346953 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.346922 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl"] Jul 28 00:04:50.355122 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.355098 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa509c8d-698fdbc56-tcttl"] Jul 28 00:04:50.565664 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.565637 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f8e69e-c733-4fcc-b39e-4b2b429d8462" path="/var/lib/kubelet/pods/92f8e69e-c733-4fcc-b39e-4b2b429d8462/volumes" Jul 28 00:04:50.566023 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.566010 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="970a0b28-f5f5-47f0-973f-bd5f176333b3" path="/var/lib/kubelet/pods/970a0b28-f5f5-47f0-973f-bd5f176333b3/volumes" Jul 28 00:04:50.566322 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.566302 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa44049d-03b4-4097-a203-dfd267b28175" path="/var/lib/kubelet/pods/aa44049d-03b4-4097-a203-dfd267b28175/volumes" Jul 28 00:04:50.951524 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:50.951431 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:54.268710 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.268673 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:54.273764 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.273727 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47"] Jul 28 00:04:54.275891 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.275869 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa509c8d-fgn47"] Jul 28 00:04:54.433599 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.433555 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:54.564899 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.564817 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd71f42b-5d36-4308-ae8c-f80b4277350e" path="/var/lib/kubelet/pods/dd71f42b-5d36-4308-ae8c-f80b4277350e/volumes" Jul 28 00:04:54.673784 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.673744 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:54.931598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:54.931507 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:55.028654 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:55.028614 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:55.139631 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:55.139596 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:55.457831 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:55.457798 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:55.774486 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:55.774450 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:57.998867 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:57.998828 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:58.071328 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071293 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9"] Jul 28 00:04:58.071878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071856 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa44049d-03b4-4097-a203-dfd267b28175" containerName="payload-pre-processing" Jul 28 00:04:58.071878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071877 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa44049d-03b4-4097-a203-dfd267b28175" containerName="payload-pre-processing" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071898 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970a0b28-f5f5-47f0-973f-bd5f176333b3" containerName="maas-api" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071907 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="970a0b28-f5f5-47f0-973f-bd5f176333b3" containerName="maas-api" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071930 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f8e69e-c733-4fcc-b39e-4b2b429d8462" containerName="payload-processing" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071939 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f8e69e-c733-4fcc-b39e-4b2b429d8462" containerName="payload-processing" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071962 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd71f42b-5d36-4308-ae8c-f80b4277350e" containerName="revoke-keys" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.071971 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd71f42b-5d36-4308-ae8c-f80b4277350e" containerName="revoke-keys" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.072044 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa44049d-03b4-4097-a203-dfd267b28175" containerName="payload-pre-processing" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.072056 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f8e69e-c733-4fcc-b39e-4b2b429d8462" containerName="payload-processing" Jul 28 00:04:58.072061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.072065 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd71f42b-5d36-4308-ae8c-f80b4277350e" containerName="revoke-keys" Jul 28 00:04:58.072584 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.072074 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="970a0b28-f5f5-47f0-973f-bd5f176333b3" containerName="maas-api" Jul 28 00:04:58.076977 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.076958 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.079216 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.079195 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-af33bc0f-serving-cert\"" Jul 28 00:04:58.085152 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.085126 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9"] Jul 28 00:04:58.104188 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.104159 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:58.123994 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.123960 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.124152 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.124014 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxthq\" (UniqueName: \"kubernetes.io/projected/01008c84-153f-4f35-922f-1921c8897b32-kube-api-access-jxthq\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.124195 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.124148 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01008c84-153f-4f35-922f-1921c8897b32-openshift-service-ca\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.225185 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.225151 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxthq\" (UniqueName: \"kubernetes.io/projected/01008c84-153f-4f35-922f-1921c8897b32-kube-api-access-jxthq\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.225366 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.225204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01008c84-153f-4f35-922f-1921c8897b32-openshift-service-ca\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.225366 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.225257 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.225366 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:04:58.225341 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-serving-cert: secret "maas-api-e2e-migrate-af33bc0f-serving-cert" not found Jul 28 00:04:58.225479 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:04:58.225394 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls podName:01008c84-153f-4f35-922f-1921c8897b32 nodeName:}" failed. No retries permitted until 2026-07-28 00:04:58.725379979 +0000 UTC m=+1698.649588835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls") pod "maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" (UID: "01008c84-153f-4f35-922f-1921c8897b32") : secret "maas-api-e2e-migrate-af33bc0f-serving-cert" not found Jul 28 00:04:58.225936 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.225914 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01008c84-153f-4f35-922f-1921c8897b32-openshift-service-ca\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.233793 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.233767 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxthq\" (UniqueName: \"kubernetes.io/projected/01008c84-153f-4f35-922f-1921c8897b32-kube-api-access-jxthq\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.311643 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.311553 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:58.617130 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.617046 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:58.716562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.716529 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:58.723829 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.723804 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d"] Jul 28 00:04:58.727328 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.727312 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.729458 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.729434 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-af33bc0f-dockercfg-wf5pk\"" Jul 28 00:04:58.729579 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.729458 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.730080 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.729932 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-af33bc0f\"" Jul 28 00:04:58.731931 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.731912 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls\") pod \"maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:58.739104 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.739080 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d"] Jul 28 00:04:58.756895 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.756871 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz"] Jul 28 00:04:58.760369 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.760350 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.768637 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.768614 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz"] Jul 28 00:04:58.830738 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.830700 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2hw\" (UniqueName: \"kubernetes.io/projected/065b8535-042a-40f2-8d50-deaea175c70e-kube-api-access-js2hw\") pod \"payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.830882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.830748 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f5d484-9973-457f-992b-bc54077080f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.830882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.830781 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b8535-042a-40f2-8d50-deaea175c70e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.830882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.830796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvwc\" (UniqueName: \"kubernetes.io/projected/60f5d484-9973-457f-992b-bc54077080f9-kube-api-access-klvwc\") pod \"payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.931572 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.931463 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2hw\" (UniqueName: \"kubernetes.io/projected/065b8535-042a-40f2-8d50-deaea175c70e-kube-api-access-js2hw\") pod \"payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.931572 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.931526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f5d484-9973-457f-992b-bc54077080f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.931572 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.931567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b8535-042a-40f2-8d50-deaea175c70e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.931870 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.931584 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvwc\" (UniqueName: \"kubernetes.io/projected/60f5d484-9973-457f-992b-bc54077080f9-kube-api-access-klvwc\") pod \"payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.932332 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.932309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f5d484-9973-457f-992b-bc54077080f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.932441 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.932308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b8535-042a-40f2-8d50-deaea175c70e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.943567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.943534 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2hw\" (UniqueName: \"kubernetes.io/projected/065b8535-042a-40f2-8d50-deaea175c70e-kube-api-access-js2hw\") pod \"payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:58.943567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.943563 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvwc\" (UniqueName: \"kubernetes.io/projected/60f5d484-9973-457f-992b-bc54077080f9-kube-api-access-klvwc\") pod \"payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:58.991323 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:58.991288 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:04:59.047758 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:59.047728 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:04:59.071370 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:59.071241 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:04:59.150620 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:59.150544 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9"] Jul 28 00:04:59.153639 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:59.153585 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01008c84_153f_4f35_922f_1921c8897b32.slice/crio-6c93743ce3e08e427addedccd14c5287a72eefc55eb4e17d555538e8b8c0d51f WatchSource:0}: Error finding container 6c93743ce3e08e427addedccd14c5287a72eefc55eb4e17d555538e8b8c0d51f: Status 404 returned error can't find the container with id 6c93743ce3e08e427addedccd14c5287a72eefc55eb4e17d555538e8b8c0d51f Jul 28 00:04:59.230781 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:59.230746 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d"] Jul 28 00:04:59.233304 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:59.233259 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065b8535_042a_40f2_8d50_deaea175c70e.slice/crio-1a627c6377e7e2273717812427f7646ac3c9b04849545ee5e59182e67c1853b5 WatchSource:0}: Error finding container 1a627c6377e7e2273717812427f7646ac3c9b04849545ee5e59182e67c1853b5: Status 404 returned error can't find the container with id 1a627c6377e7e2273717812427f7646ac3c9b04849545ee5e59182e67c1853b5 Jul 28 00:04:59.456660 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:04:59.456634 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz"] Jul 28 00:04:59.458421 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:04:59.458393 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f5d484_9973_457f_992b_bc54077080f9.slice/crio-0f54b385bb1f5ad8a8fcdf605c99afe065393bcf0bbd0a4090690fe5ec6d4dcf WatchSource:0}: Error finding container 0f54b385bb1f5ad8a8fcdf605c99afe065393bcf0bbd0a4090690fe5ec6d4dcf: Status 404 returned error can't find the container with id 0f54b385bb1f5ad8a8fcdf605c99afe065393bcf0bbd0a4090690fe5ec6d4dcf Jul 28 00:05:00.082471 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.082305 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" event={"ID":"60f5d484-9973-457f-992b-bc54077080f9","Type":"ContainerStarted","Data":"d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae"} Jul 28 00:05:00.082471 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.082413 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:05:00.082471 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.082440 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" event={"ID":"60f5d484-9973-457f-992b-bc54077080f9","Type":"ContainerStarted","Data":"0f54b385bb1f5ad8a8fcdf605c99afe065393bcf0bbd0a4090690fe5ec6d4dcf"} Jul 28 00:05:00.084828 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.084377 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" event={"ID":"065b8535-042a-40f2-8d50-deaea175c70e","Type":"ContainerStarted","Data":"66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d"} Jul 28 00:05:00.084828 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.084452 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:05:00.084828 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.084474 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" event={"ID":"065b8535-042a-40f2-8d50-deaea175c70e","Type":"ContainerStarted","Data":"1a627c6377e7e2273717812427f7646ac3c9b04849545ee5e59182e67c1853b5"} Jul 28 00:05:00.086577 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.086535 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" event={"ID":"01008c84-153f-4f35-922f-1921c8897b32","Type":"ContainerStarted","Data":"b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43"} Jul 28 00:05:00.086718 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.086582 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" event={"ID":"01008c84-153f-4f35-922f-1921c8897b32","Type":"ContainerStarted","Data":"6c93743ce3e08e427addedccd14c5287a72eefc55eb4e17d555538e8b8c0d51f"} Jul 28 00:05:00.087264 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.087231 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:05:00.097957 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.097900 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" podStartSLOduration=2.097876879 podStartE2EDuration="2.097876879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:00.09700583 +0000 UTC m=+1700.021214711" watchObservedRunningTime="2026-07-28 00:05:00.097876879 +0000 UTC m=+1700.022085751" Jul 28 00:05:00.113881 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.113816 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" podStartSLOduration=1.712048668 podStartE2EDuration="2.113800175s" totalImagesPullingTime="401.751507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:00.111507094 +0000 UTC m=+1700.035715966" watchObservedRunningTime="2026-07-28 00:05:00.113800175 +0000 UTC m=+1700.038009053" Jul 28 00:05:00.125271 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:00.125219 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" podStartSLOduration=2.125200717 podStartE2EDuration="2.125200717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:00.124453353 +0000 UTC m=+1700.048662231" watchObservedRunningTime="2026-07-28 00:05:00.125200717 +0000 UTC m=+1700.049409597" Jul 28 00:05:07.105136 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:07.105061 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:05:11.096098 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:11.096061 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:05:11.096616 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:11.096235 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:05:23.002017 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.001979 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s"] Jul 28 00:05:23.006959 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.006930 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:23.010032 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.009991 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:05:23.011036 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.011010 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s"] Jul 28 00:05:23.153800 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.153765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f394c8-7eca-4266-a156-eb81357135c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s\" (UID: \"a0f394c8-7eca-4266-a156-eb81357135c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:23.254955 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.254878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f394c8-7eca-4266-a156-eb81357135c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s\" (UID: \"a0f394c8-7eca-4266-a156-eb81357135c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:23.255588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.255489 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f394c8-7eca-4266-a156-eb81357135c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s\" (UID: \"a0f394c8-7eca-4266-a156-eb81357135c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:23.319949 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.319920 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:23.661126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:23.661103 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s"] Jul 28 00:05:23.663157 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:05:23.663131 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f394c8_7eca_4266_a156_eb81357135c9.slice/crio-faca75631e74798691ec0a7942437e2a4c1e3f953e7e63c8acfd35dd1b269d24 WatchSource:0}: Error finding container faca75631e74798691ec0a7942437e2a4c1e3f953e7e63c8acfd35dd1b269d24: Status 404 returned error can't find the container with id faca75631e74798691ec0a7942437e2a4c1e3f953e7e63c8acfd35dd1b269d24 Jul 28 00:05:24.210043 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:24.209955 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" event={"ID":"a0f394c8-7eca-4266-a156-eb81357135c9","Type":"ContainerStarted","Data":"6d44066c5c0863dc5ef35552186300a6903aa8ad713d40282145cec27f1c1597"} Jul 28 00:05:24.210043 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:24.210032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" event={"ID":"a0f394c8-7eca-4266-a156-eb81357135c9","Type":"ContainerStarted","Data":"faca75631e74798691ec0a7942437e2a4c1e3f953e7e63c8acfd35dd1b269d24"} Jul 28 00:05:24.224827 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:24.224782 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" podStartSLOduration=2.224768595 podStartE2EDuration="2.224768595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:24.223052139 +0000 UTC m=+1724.147261038" watchObservedRunningTime="2026-07-28 00:05:24.224768595 +0000 UTC m=+1724.148977494" Jul 28 00:05:25.216567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:25.216535 2555 generic.go:356] "Generic (PLEG): container finished" podID="a0f394c8-7eca-4266-a156-eb81357135c9" containerID="6d44066c5c0863dc5ef35552186300a6903aa8ad713d40282145cec27f1c1597" exitCode=0 Jul 28 00:05:25.216951 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:25.216625 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" event={"ID":"a0f394c8-7eca-4266-a156-eb81357135c9","Type":"ContainerDied","Data":"6d44066c5c0863dc5ef35552186300a6903aa8ad713d40282145cec27f1c1597"} Jul 28 00:05:26.353734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:26.353709 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:26.490298 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:26.490277 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f394c8-7eca-4266-a156-eb81357135c9-maas-api-service-ca\") pod \"a0f394c8-7eca-4266-a156-eb81357135c9\" (UID: \"a0f394c8-7eca-4266-a156-eb81357135c9\") " Jul 28 00:05:26.490652 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:26.490630 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f394c8-7eca-4266-a156-eb81357135c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0f394c8-7eca-4266-a156-eb81357135c9" (UID: "a0f394c8-7eca-4266-a156-eb81357135c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:05:26.591324 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:26.591292 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f394c8-7eca-4266-a156-eb81357135c9-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:27.229686 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:27.229650 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" event={"ID":"a0f394c8-7eca-4266-a156-eb81357135c9","Type":"ContainerDied","Data":"faca75631e74798691ec0a7942437e2a4c1e3f953e7e63c8acfd35dd1b269d24"} Jul 28 00:05:27.229686 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:27.229685 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s" Jul 28 00:05:27.229896 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:27.229699 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faca75631e74798691ec0a7942437e2a4c1e3f953e7e63c8acfd35dd1b269d24" Jul 28 00:05:33.385803 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.385765 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9"] Jul 28 00:05:33.386285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.386020 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" podUID="01008c84-153f-4f35-922f-1921c8897b32" containerName="maas-api" containerID="cri-o://b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43" gracePeriod=30 Jul 28 00:05:33.407767 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.407735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:33.437871 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.437841 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz"] Jul 28 00:05:33.438069 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.438043 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" podUID="60f5d484-9973-457f-992b-bc54077080f9" containerName="payload-processing" containerID="cri-o://d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae" gracePeriod=30 Jul 28 00:05:33.445161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.445137 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d"] Jul 28 00:05:33.445565 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.445540 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" podUID="065b8535-042a-40f2-8d50-deaea175c70e" containerName="payload-pre-processing" containerID="cri-o://66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d" gracePeriod=30 Jul 28 00:05:33.697733 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.697697 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:05:33.738389 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.738355 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:05:33.745583 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.745560 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:05:33.746398 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746381 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01008c84-153f-4f35-922f-1921c8897b32-openshift-service-ca\") pod \"01008c84-153f-4f35-922f-1921c8897b32\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " Jul 28 00:05:33.746485 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746415 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b8535-042a-40f2-8d50-deaea175c70e-plugins-config-volume\") pod \"065b8535-042a-40f2-8d50-deaea175c70e\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " Jul 28 00:05:33.746567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746521 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klvwc\" (UniqueName: \"kubernetes.io/projected/60f5d484-9973-457f-992b-bc54077080f9-kube-api-access-klvwc\") pod \"60f5d484-9973-457f-992b-bc54077080f9\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " Jul 28 00:05:33.746567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746558 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls\") pod \"01008c84-153f-4f35-922f-1921c8897b32\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " Jul 28 00:05:33.746697 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746680 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxthq\" (UniqueName: \"kubernetes.io/projected/01008c84-153f-4f35-922f-1921c8897b32-kube-api-access-jxthq\") pod \"01008c84-153f-4f35-922f-1921c8897b32\" (UID: \"01008c84-153f-4f35-922f-1921c8897b32\") " Jul 28 00:05:33.746754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746708 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01008c84-153f-4f35-922f-1921c8897b32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01008c84-153f-4f35-922f-1921c8897b32" (UID: "01008c84-153f-4f35-922f-1921c8897b32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:05:33.746754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746728 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js2hw\" (UniqueName: \"kubernetes.io/projected/065b8535-042a-40f2-8d50-deaea175c70e-kube-api-access-js2hw\") pod \"065b8535-042a-40f2-8d50-deaea175c70e\" (UID: \"065b8535-042a-40f2-8d50-deaea175c70e\") " Jul 28 00:05:33.746861 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746757 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f5d484-9973-457f-992b-bc54077080f9-plugins-config-volume\") pod \"60f5d484-9973-457f-992b-bc54077080f9\" (UID: \"60f5d484-9973-457f-992b-bc54077080f9\") " Jul 28 00:05:33.746941 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.746914 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065b8535-042a-40f2-8d50-deaea175c70e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "065b8535-042a-40f2-8d50-deaea175c70e" (UID: "065b8535-042a-40f2-8d50-deaea175c70e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:05:33.747109 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.747085 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01008c84-153f-4f35-922f-1921c8897b32-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.747219 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.747111 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b8535-042a-40f2-8d50-deaea175c70e-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.747219 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.747185 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f5d484-9973-457f-992b-bc54077080f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60f5d484-9973-457f-992b-bc54077080f9" (UID: "60f5d484-9973-457f-992b-bc54077080f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:05:33.748574 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.748552 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f5d484-9973-457f-992b-bc54077080f9-kube-api-access-klvwc" (OuterVolumeSpecName: "kube-api-access-klvwc") pod "60f5d484-9973-457f-992b-bc54077080f9" (UID: "60f5d484-9973-457f-992b-bc54077080f9"). InnerVolumeSpecName "kube-api-access-klvwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:05:33.748670 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.748654 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01008c84-153f-4f35-922f-1921c8897b32" (UID: "01008c84-153f-4f35-922f-1921c8897b32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:05:33.749099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.749071 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065b8535-042a-40f2-8d50-deaea175c70e-kube-api-access-js2hw" (OuterVolumeSpecName: "kube-api-access-js2hw") pod "065b8535-042a-40f2-8d50-deaea175c70e" (UID: "065b8535-042a-40f2-8d50-deaea175c70e"). InnerVolumeSpecName "kube-api-access-js2hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:05:33.749176 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.749102 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01008c84-153f-4f35-922f-1921c8897b32-kube-api-access-jxthq" (OuterVolumeSpecName: "kube-api-access-jxthq") pod "01008c84-153f-4f35-922f-1921c8897b32" (UID: "01008c84-153f-4f35-922f-1921c8897b32"). InnerVolumeSpecName "kube-api-access-jxthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:05:33.847505 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.847467 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klvwc\" (UniqueName: \"kubernetes.io/projected/60f5d484-9973-457f-992b-bc54077080f9-kube-api-access-klvwc\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.847671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.847525 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01008c84-153f-4f35-922f-1921c8897b32-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.847671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.847536 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxthq\" (UniqueName: \"kubernetes.io/projected/01008c84-153f-4f35-922f-1921c8897b32-kube-api-access-jxthq\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.847671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.847545 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2hw\" (UniqueName: \"kubernetes.io/projected/065b8535-042a-40f2-8d50-deaea175c70e-kube-api-access-js2hw\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.847671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.847554 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f5d484-9973-457f-992b-bc54077080f9-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:05:33.917945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.917849 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:33.981717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:33.981682 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:34.264300 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.264267 2555 generic.go:356] "Generic (PLEG): container finished" podID="60f5d484-9973-457f-992b-bc54077080f9" containerID="d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae" exitCode=0 Jul 28 00:05:34.264481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.264314 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" Jul 28 00:05:34.264481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.264355 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" event={"ID":"60f5d484-9973-457f-992b-bc54077080f9","Type":"ContainerDied","Data":"d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae"} Jul 28 00:05:34.264481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.264414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz" event={"ID":"60f5d484-9973-457f-992b-bc54077080f9","Type":"ContainerDied","Data":"0f54b385bb1f5ad8a8fcdf605c99afe065393bcf0bbd0a4090690fe5ec6d4dcf"} Jul 28 00:05:34.264481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.264444 2555 scope.go:117] "RemoveContainer" containerID="d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae" Jul 28 00:05:34.265878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.265854 2555 generic.go:356] "Generic (PLEG): container finished" podID="065b8535-042a-40f2-8d50-deaea175c70e" containerID="66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d" exitCode=0 Jul 28 00:05:34.265988 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.265902 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" Jul 28 00:05:34.266048 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.265897 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" event={"ID":"065b8535-042a-40f2-8d50-deaea175c70e","Type":"ContainerDied","Data":"66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d"} Jul 28 00:05:34.266110 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.266044 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d" event={"ID":"065b8535-042a-40f2-8d50-deaea175c70e","Type":"ContainerDied","Data":"1a627c6377e7e2273717812427f7646ac3c9b04849545ee5e59182e67c1853b5"} Jul 28 00:05:34.269916 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.269884 2555 generic.go:356] "Generic (PLEG): container finished" podID="01008c84-153f-4f35-922f-1921c8897b32" containerID="b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43" exitCode=0 Jul 28 00:05:34.270020 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.269939 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" event={"ID":"01008c84-153f-4f35-922f-1921c8897b32","Type":"ContainerDied","Data":"b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43"} Jul 28 00:05:34.270090 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.270029 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" event={"ID":"01008c84-153f-4f35-922f-1921c8897b32","Type":"ContainerDied","Data":"6c93743ce3e08e427addedccd14c5287a72eefc55eb4e17d555538e8b8c0d51f"} Jul 28 00:05:34.270152 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.270141 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9" Jul 28 00:05:34.281445 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.281429 2555 scope.go:117] "RemoveContainer" containerID="d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae" Jul 28 00:05:34.282033 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:05:34.282010 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae\": container with ID starting with d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae not found: ID does not exist" containerID="d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae" Jul 28 00:05:34.282133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.282041 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae"} err="failed to get container status \"d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae\": rpc error: code = NotFound desc = could not find container \"d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae\": container with ID starting with d03a4838e7ec913ee6fa1189f5c80cdb8293a8a9dd3da2f2b982c28db7bf1eae not found: ID does not exist" Jul 28 00:05:34.282133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.282063 2555 scope.go:117] "RemoveContainer" containerID="66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d" Jul 28 00:05:34.290291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.290273 2555 scope.go:117] "RemoveContainer" containerID="66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d" Jul 28 00:05:34.290590 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:05:34.290569 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d\": container with ID starting with 66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d not found: ID does not exist" containerID="66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d" Jul 28 00:05:34.290665 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.290596 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d"} err="failed to get container status \"66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d\": rpc error: code = NotFound desc = could not find container \"66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d\": container with ID starting with 66cbe3ca0a6cdc8edfb33453e8267c9d9dadc684d586fd464112a8b121910b5d not found: ID does not exist" Jul 28 00:05:34.290665 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.290612 2555 scope.go:117] "RemoveContainer" containerID="b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43" Jul 28 00:05:34.292481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.292462 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz"] Jul 28 00:05:34.296143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.296116 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-af33bc0f-58b75995fc-h6kgz"] Jul 28 00:05:34.298393 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.298377 2555 scope.go:117] "RemoveContainer" containerID="b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43" Jul 28 00:05:34.298646 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:05:34.298627 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43\": container with ID starting with b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43 not found: ID does not exist" containerID="b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43" Jul 28 00:05:34.298711 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.298652 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43"} err="failed to get container status \"b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43\": rpc error: code = NotFound desc = could not find container \"b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43\": container with ID starting with b0f810ea7c265e1a79af3891f72ca9809b953c7f791f20a53128d662890a9e43 not found: ID does not exist" Jul 28 00:05:34.304783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.304763 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9"] Jul 28 00:05:34.309877 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.309858 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af33bc0f-6d8965f57-tdhx9"] Jul 28 00:05:34.318517 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.318477 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d"] Jul 28 00:05:34.320153 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.320135 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-af33bc0f-649b556776-k2n8d"] Jul 28 00:05:34.565381 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.565300 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01008c84-153f-4f35-922f-1921c8897b32" path="/var/lib/kubelet/pods/01008c84-153f-4f35-922f-1921c8897b32/volumes" Jul 28 00:05:34.565734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.565670 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065b8535-042a-40f2-8d50-deaea175c70e" path="/var/lib/kubelet/pods/065b8535-042a-40f2-8d50-deaea175c70e/volumes" Jul 28 00:05:34.565973 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:34.565961 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f5d484-9973-457f-992b-bc54077080f9" path="/var/lib/kubelet/pods/60f5d484-9973-457f-992b-bc54077080f9/volumes" Jul 28 00:05:38.076188 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.076152 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s"] Jul 28 00:05:38.078975 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.078952 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af33bc0f-b4q7s"] Jul 28 00:05:38.082698 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.082655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:38.242483 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.242449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:38.482273 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.482190 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:38.565068 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.565038 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0f394c8-7eca-4266-a156-eb81357135c9" path="/var/lib/kubelet/pods/a0f394c8-7eca-4266-a156-eb81357135c9/volumes" Jul 28 00:05:38.720007 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:38.719978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:39.031895 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:39.031861 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:39.502514 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:39.502472 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:41.117416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.117328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:41.219555 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.219520 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:41.224633 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.224595 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x"] Jul 28 00:05:41.225273 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225243 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065b8535-042a-40f2-8d50-deaea175c70e" containerName="payload-pre-processing" Jul 28 00:05:41.225273 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225271 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="065b8535-042a-40f2-8d50-deaea175c70e" containerName="payload-pre-processing" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225294 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f5d484-9973-457f-992b-bc54077080f9" containerName="payload-processing" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225303 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f5d484-9973-457f-992b-bc54077080f9" containerName="payload-processing" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225316 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01008c84-153f-4f35-922f-1921c8897b32" containerName="maas-api" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225326 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="01008c84-153f-4f35-922f-1921c8897b32" containerName="maas-api" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225345 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f394c8-7eca-4266-a156-eb81357135c9" containerName="revoke-keys" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225354 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f394c8-7eca-4266-a156-eb81357135c9" containerName="revoke-keys" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225442 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f394c8-7eca-4266-a156-eb81357135c9" containerName="revoke-keys" Jul 28 00:05:41.225454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225456 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f5d484-9973-457f-992b-bc54077080f9" containerName="payload-processing" Jul 28 00:05:41.225837 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225468 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="01008c84-153f-4f35-922f-1921c8897b32" containerName="maas-api" Jul 28 00:05:41.225837 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.225481 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="065b8535-042a-40f2-8d50-deaea175c70e" containerName="payload-pre-processing" Jul 28 00:05:41.229711 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.229689 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.231632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.231614 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1a3da35c-serving-cert\"" Jul 28 00:05:41.239932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.239859 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x"] Jul 28 00:05:41.303597 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.303537 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0835a438-de38-464f-b3cc-cccea46fcaad-openshift-service-ca\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.303792 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.303754 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh572\" (UniqueName: \"kubernetes.io/projected/0835a438-de38-464f-b3cc-cccea46fcaad-kube-api-access-lh572\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.303892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.303871 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0835a438-de38-464f-b3cc-cccea46fcaad-maas-api-tls\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.321006 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.320975 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:41.404589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.404485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0835a438-de38-464f-b3cc-cccea46fcaad-openshift-service-ca\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.404589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.404547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh572\" (UniqueName: \"kubernetes.io/projected/0835a438-de38-464f-b3cc-cccea46fcaad-kube-api-access-lh572\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.404765 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.404600 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0835a438-de38-464f-b3cc-cccea46fcaad-maas-api-tls\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.405172 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.405147 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0835a438-de38-464f-b3cc-cccea46fcaad-openshift-service-ca\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.407086 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.407060 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0835a438-de38-464f-b3cc-cccea46fcaad-maas-api-tls\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.411737 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.411714 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh572\" (UniqueName: \"kubernetes.io/projected/0835a438-de38-464f-b3cc-cccea46fcaad-kube-api-access-lh572\") pod \"maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.423614 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.423593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:41.543017 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.542987 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:41.673931 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.673904 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x"] Jul 28 00:05:41.675842 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:05:41.675810 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0835a438_de38_464f_b3cc_cccea46fcaad.slice/crio-4c595fd393eb927ca3709f88f50212e27c28bee2203c21a8bc303900e5e5e1b5 WatchSource:0}: Error finding container 4c595fd393eb927ca3709f88f50212e27c28bee2203c21a8bc303900e5e5e1b5: Status 404 returned error can't find the container with id 4c595fd393eb927ca3709f88f50212e27c28bee2203c21a8bc303900e5e5e1b5 Jul 28 00:05:41.879920 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.879879 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:41.988877 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.988792 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s"] Jul 28 00:05:41.991849 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.991833 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:41.997530 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.997484 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1a3da35c\"" Jul 28 00:05:41.997651 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:41.997530 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1a3da35c-dockercfg-xv2sh\"" Jul 28 00:05:42.004531 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.004490 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s"] Jul 28 00:05:42.009680 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.009653 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb609ea-1c4d-477b-a772-32de0cb8372b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.009841 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.009821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5qd\" (UniqueName: \"kubernetes.io/projected/3cb609ea-1c4d-477b-a772-32de0cb8372b-kube-api-access-6n5qd\") pod \"payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.017762 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.017741 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt"] Jul 28 00:05:42.020296 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.020281 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.028568 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.028545 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt"] Jul 28 00:05:42.110709 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.110673 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.110882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.110721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5qd\" (UniqueName: \"kubernetes.io/projected/3cb609ea-1c4d-477b-a772-32de0cb8372b-kube-api-access-6n5qd\") pod \"payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.110882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.110746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbbh\" (UniqueName: \"kubernetes.io/projected/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-kube-api-access-nxbbh\") pod \"payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.110882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.110766 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb609ea-1c4d-477b-a772-32de0cb8372b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.111477 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.111453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb609ea-1c4d-477b-a772-32de0cb8372b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.118353 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.118329 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5qd\" (UniqueName: \"kubernetes.io/projected/3cb609ea-1c4d-477b-a772-32de0cb8372b-kube-api-access-6n5qd\") pod \"payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.211942 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.211899 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.212113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.211979 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbbh\" (UniqueName: \"kubernetes.io/projected/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-kube-api-access-nxbbh\") pod \"payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.212669 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.212649 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.219111 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.219079 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbbh\" (UniqueName: \"kubernetes.io/projected/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-kube-api-access-nxbbh\") pod \"payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.301703 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.301659 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:42.301888 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.301860 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:05:42.319145 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.319119 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" event={"ID":"0835a438-de38-464f-b3cc-cccea46fcaad","Type":"ContainerStarted","Data":"4c595fd393eb927ca3709f88f50212e27c28bee2203c21a8bc303900e5e5e1b5"} Jul 28 00:05:42.332287 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.332263 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:42.450381 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.450353 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s"] Jul 28 00:05:42.453133 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:05:42.453102 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb609ea_1c4d_477b_a772_32de0cb8372b.slice/crio-03fb08204e8dea48d2b57771c87d433cdd92e0191953c578431e9988585b08a3 WatchSource:0}: Error finding container 03fb08204e8dea48d2b57771c87d433cdd92e0191953c578431e9988585b08a3: Status 404 returned error can't find the container with id 03fb08204e8dea48d2b57771c87d433cdd92e0191953c578431e9988585b08a3 Jul 28 00:05:42.690720 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:42.690476 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt"] Jul 28 00:05:42.693164 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:05:42.693132 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27efbd5_545e_4ef5_bad7_3ae6cb0ecf56.slice/crio-dfb29c111746eee52cc8d52815878e4f594180a92863f3c5310deffa5c0fbd87 WatchSource:0}: Error finding container dfb29c111746eee52cc8d52815878e4f594180a92863f3c5310deffa5c0fbd87: Status 404 returned error can't find the container with id dfb29c111746eee52cc8d52815878e4f594180a92863f3c5310deffa5c0fbd87 Jul 28 00:05:43.326736 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.326695 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" event={"ID":"3cb609ea-1c4d-477b-a772-32de0cb8372b","Type":"ContainerStarted","Data":"8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a"} Jul 28 00:05:43.327203 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.326764 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:05:43.327203 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.326789 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" event={"ID":"3cb609ea-1c4d-477b-a772-32de0cb8372b","Type":"ContainerStarted","Data":"03fb08204e8dea48d2b57771c87d433cdd92e0191953c578431e9988585b08a3"} Jul 28 00:05:43.328293 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.328261 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" event={"ID":"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56","Type":"ContainerStarted","Data":"2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480"} Jul 28 00:05:43.328433 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.328318 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:43.328433 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.328343 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" event={"ID":"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56","Type":"ContainerStarted","Data":"dfb29c111746eee52cc8d52815878e4f594180a92863f3c5310deffa5c0fbd87"} Jul 28 00:05:43.330069 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.330044 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" event={"ID":"0835a438-de38-464f-b3cc-cccea46fcaad","Type":"ContainerStarted","Data":"4244851249fedb0b11284f12ec6e5af10c86fd8a1c5ef2c7ad32c532746f7459"} Jul 28 00:05:43.330177 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.330164 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:43.340171 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.340127 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" podStartSLOduration=2.340112072 podStartE2EDuration="2.340112072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:43.33972206 +0000 UTC m=+1743.263930938" watchObservedRunningTime="2026-07-28 00:05:43.340112072 +0000 UTC m=+1743.264320951" Jul 28 00:05:43.354962 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.354926 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" podStartSLOduration=1.512121241 podStartE2EDuration="2.354915196s" totalImagesPullingTime="842.793955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:43.353264862 +0000 UTC m=+1743.277473740" watchObservedRunningTime="2026-07-28 00:05:43.354915196 +0000 UTC m=+1743.279124074" Jul 28 00:05:43.368553 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:43.368516 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" podStartSLOduration=1.368490654 podStartE2EDuration="1.368490654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:05:43.366999365 +0000 UTC m=+1743.291208243" watchObservedRunningTime="2026-07-28 00:05:43.368490654 +0000 UTC m=+1743.292699532" Jul 28 00:05:49.339937 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:49.339903 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:05:54.336911 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:54.336874 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:05:54.337433 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:05:54.337085 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:06:06.935608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:06.935569 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr"] Jul 28 00:06:06.940297 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:06.940280 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:06.942367 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:06.942333 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:06:06.944960 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:06.944937 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr"] Jul 28 00:06:07.040581 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:07.040541 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae560bf6-7023-4dd2-ba10-8b396cda4a36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr\" (UID: \"ae560bf6-7023-4dd2-ba10-8b396cda4a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:07.141785 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:07.141750 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae560bf6-7023-4dd2-ba10-8b396cda4a36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr\" (UID: \"ae560bf6-7023-4dd2-ba10-8b396cda4a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:07.142372 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:07.142352 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae560bf6-7023-4dd2-ba10-8b396cda4a36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr\" (UID: \"ae560bf6-7023-4dd2-ba10-8b396cda4a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:07.254398 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:07.254369 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:07.385380 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:07.385350 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr"] Jul 28 00:06:07.386224 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:06:07.386197 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae560bf6_7023_4dd2_ba10_8b396cda4a36.slice/crio-87db18224076b677d8e76ffede4797ba31c057869724525d3a6360376be09064 WatchSource:0}: Error finding container 87db18224076b677d8e76ffede4797ba31c057869724525d3a6360376be09064: Status 404 returned error can't find the container with id 87db18224076b677d8e76ffede4797ba31c057869724525d3a6360376be09064 Jul 28 00:06:07.449407 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:07.449377 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" event={"ID":"ae560bf6-7023-4dd2-ba10-8b396cda4a36","Type":"ContainerStarted","Data":"87db18224076b677d8e76ffede4797ba31c057869724525d3a6360376be09064"} Jul 28 00:06:08.456207 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:08.456128 2555 generic.go:356] "Generic (PLEG): container finished" podID="ae560bf6-7023-4dd2-ba10-8b396cda4a36" containerID="f6e35cb508fd9000f08bf14f224a6ea444b687d035afafbc3fbd1927a6ea723c" exitCode=0 Jul 28 00:06:08.456612 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:08.456208 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" event={"ID":"ae560bf6-7023-4dd2-ba10-8b396cda4a36","Type":"ContainerDied","Data":"f6e35cb508fd9000f08bf14f224a6ea444b687d035afafbc3fbd1927a6ea723c"} Jul 28 00:06:09.596107 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:09.596086 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:09.664455 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:09.664422 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae560bf6-7023-4dd2-ba10-8b396cda4a36-maas-api-service-ca\") pod \"ae560bf6-7023-4dd2-ba10-8b396cda4a36\" (UID: \"ae560bf6-7023-4dd2-ba10-8b396cda4a36\") " Jul 28 00:06:09.664834 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:09.664807 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae560bf6-7023-4dd2-ba10-8b396cda4a36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae560bf6-7023-4dd2-ba10-8b396cda4a36" (UID: "ae560bf6-7023-4dd2-ba10-8b396cda4a36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:06:09.765561 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:09.765526 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae560bf6-7023-4dd2-ba10-8b396cda4a36-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:10.467095 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:10.467063 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" Jul 28 00:06:10.467263 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:10.467073 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr" event={"ID":"ae560bf6-7023-4dd2-ba10-8b396cda4a36","Type":"ContainerDied","Data":"87db18224076b677d8e76ffede4797ba31c057869724525d3a6360376be09064"} Jul 28 00:06:10.467263 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:10.467179 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87db18224076b677d8e76ffede4797ba31c057869724525d3a6360376be09064" Jul 28 00:06:18.360938 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.360903 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x"] Jul 28 00:06:18.361439 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.361191 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" podUID="0835a438-de38-464f-b3cc-cccea46fcaad" containerName="maas-api" containerID="cri-o://4244851249fedb0b11284f12ec6e5af10c86fd8a1c5ef2c7ad32c532746f7459" gracePeriod=30 Jul 28 00:06:18.412804 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.412310 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt"] Jul 28 00:06:18.412804 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.412486 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s"] Jul 28 00:06:18.413036 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.412819 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" podUID="3cb609ea-1c4d-477b-a772-32de0cb8372b" containerName="payload-pre-processing" containerID="cri-o://8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a" gracePeriod=30 Jul 28 00:06:18.413776 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.413135 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" podUID="e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" containerName="payload-processing" containerID="cri-o://2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480" gracePeriod=30 Jul 28 00:06:18.446369 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.446103 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:18.514682 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.514653 2555 generic.go:356] "Generic (PLEG): container finished" podID="0835a438-de38-464f-b3cc-cccea46fcaad" containerID="4244851249fedb0b11284f12ec6e5af10c86fd8a1c5ef2c7ad32c532746f7459" exitCode=0 Jul 28 00:06:18.514827 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.514734 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" event={"ID":"0835a438-de38-464f-b3cc-cccea46fcaad","Type":"ContainerDied","Data":"4244851249fedb0b11284f12ec6e5af10c86fd8a1c5ef2c7ad32c532746f7459"} Jul 28 00:06:18.545291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.545252 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:18.703805 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.703781 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:06:18.709080 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.709054 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:06:18.724801 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.724772 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:06:18.843967 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.843935 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0835a438-de38-464f-b3cc-cccea46fcaad-maas-api-tls\") pod \"0835a438-de38-464f-b3cc-cccea46fcaad\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " Jul 28 00:06:18.844131 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844023 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0835a438-de38-464f-b3cc-cccea46fcaad-openshift-service-ca\") pod \"0835a438-de38-464f-b3cc-cccea46fcaad\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " Jul 28 00:06:18.844175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844123 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb609ea-1c4d-477b-a772-32de0cb8372b-plugins-config-volume\") pod \"3cb609ea-1c4d-477b-a772-32de0cb8372b\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " Jul 28 00:06:18.844175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844168 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh572\" (UniqueName: \"kubernetes.io/projected/0835a438-de38-464f-b3cc-cccea46fcaad-kube-api-access-lh572\") pod \"0835a438-de38-464f-b3cc-cccea46fcaad\" (UID: \"0835a438-de38-464f-b3cc-cccea46fcaad\") " Jul 28 00:06:18.844262 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844215 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-plugins-config-volume\") pod \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " Jul 28 00:06:18.844262 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844233 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n5qd\" (UniqueName: \"kubernetes.io/projected/3cb609ea-1c4d-477b-a772-32de0cb8372b-kube-api-access-6n5qd\") pod \"3cb609ea-1c4d-477b-a772-32de0cb8372b\" (UID: \"3cb609ea-1c4d-477b-a772-32de0cb8372b\") " Jul 28 00:06:18.844262 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844252 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxbbh\" (UniqueName: \"kubernetes.io/projected/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-kube-api-access-nxbbh\") pod \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\" (UID: \"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56\") " Jul 28 00:06:18.844589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844562 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb609ea-1c4d-477b-a772-32de0cb8372b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cb609ea-1c4d-477b-a772-32de0cb8372b" (UID: "3cb609ea-1c4d-477b-a772-32de0cb8372b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:06:18.844715 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844591 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" (UID: "e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:06:18.844796 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.844763 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0835a438-de38-464f-b3cc-cccea46fcaad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0835a438-de38-464f-b3cc-cccea46fcaad" (UID: "0835a438-de38-464f-b3cc-cccea46fcaad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:06:18.846332 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.846302 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0835a438-de38-464f-b3cc-cccea46fcaad-kube-api-access-lh572" (OuterVolumeSpecName: "kube-api-access-lh572") pod "0835a438-de38-464f-b3cc-cccea46fcaad" (UID: "0835a438-de38-464f-b3cc-cccea46fcaad"). InnerVolumeSpecName "kube-api-access-lh572". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:06:18.846416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.846339 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0835a438-de38-464f-b3cc-cccea46fcaad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0835a438-de38-464f-b3cc-cccea46fcaad" (UID: "0835a438-de38-464f-b3cc-cccea46fcaad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:06:18.846416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.846352 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb609ea-1c4d-477b-a772-32de0cb8372b-kube-api-access-6n5qd" (OuterVolumeSpecName: "kube-api-access-6n5qd") pod "3cb609ea-1c4d-477b-a772-32de0cb8372b" (UID: "3cb609ea-1c4d-477b-a772-32de0cb8372b"). InnerVolumeSpecName "kube-api-access-6n5qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:06:18.846416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.846369 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-kube-api-access-nxbbh" (OuterVolumeSpecName: "kube-api-access-nxbbh") pod "e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" (UID: "e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56"). InnerVolumeSpecName "kube-api-access-nxbbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945046 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0835a438-de38-464f-b3cc-cccea46fcaad-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945073 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb609ea-1c4d-477b-a772-32de0cb8372b-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945082 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh572\" (UniqueName: \"kubernetes.io/projected/0835a438-de38-464f-b3cc-cccea46fcaad-kube-api-access-lh572\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945091 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945099 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5qd\" (UniqueName: \"kubernetes.io/projected/3cb609ea-1c4d-477b-a772-32de0cb8372b-kube-api-access-6n5qd\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945107 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbbh\" (UniqueName: \"kubernetes.io/projected/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56-kube-api-access-nxbbh\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:18.945116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:18.945116 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0835a438-de38-464f-b3cc-cccea46fcaad-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:19.521161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.521122 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb609ea-1c4d-477b-a772-32de0cb8372b" containerID="8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a" exitCode=0 Jul 28 00:06:19.521161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.521164 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" Jul 28 00:06:19.521696 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.521213 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" event={"ID":"3cb609ea-1c4d-477b-a772-32de0cb8372b","Type":"ContainerDied","Data":"8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a"} Jul 28 00:06:19.521696 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.521261 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s" event={"ID":"3cb609ea-1c4d-477b-a772-32de0cb8372b","Type":"ContainerDied","Data":"03fb08204e8dea48d2b57771c87d433cdd92e0191953c578431e9988585b08a3"} Jul 28 00:06:19.521696 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.521291 2555 scope.go:117] "RemoveContainer" containerID="8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a" Jul 28 00:06:19.522592 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.522573 2555 generic.go:356] "Generic (PLEG): container finished" podID="e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" containerID="2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480" exitCode=0 Jul 28 00:06:19.522670 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.522646 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" Jul 28 00:06:19.522731 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.522706 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" event={"ID":"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56","Type":"ContainerDied","Data":"2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480"} Jul 28 00:06:19.522782 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.522758 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt" event={"ID":"e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56","Type":"ContainerDied","Data":"dfb29c111746eee52cc8d52815878e4f594180a92863f3c5310deffa5c0fbd87"} Jul 28 00:06:19.524067 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.524043 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" event={"ID":"0835a438-de38-464f-b3cc-cccea46fcaad","Type":"ContainerDied","Data":"4c595fd393eb927ca3709f88f50212e27c28bee2203c21a8bc303900e5e5e1b5"} Jul 28 00:06:19.524167 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.524074 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x" Jul 28 00:06:19.530653 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.530638 2555 scope.go:117] "RemoveContainer" containerID="8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a" Jul 28 00:06:19.530899 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:06:19.530878 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a\": container with ID starting with 8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a not found: ID does not exist" containerID="8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a" Jul 28 00:06:19.530970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.530909 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a"} err="failed to get container status \"8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a\": rpc error: code = NotFound desc = could not find container \"8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a\": container with ID starting with 8ef2adfd06b089ba03572be6ce67ab8c2a98c4e8c05c7ff7683d587aa964790a not found: ID does not exist" Jul 28 00:06:19.530970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.530927 2555 scope.go:117] "RemoveContainer" containerID="2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480" Jul 28 00:06:19.539966 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.539948 2555 scope.go:117] "RemoveContainer" containerID="2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480" Jul 28 00:06:19.540221 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:06:19.540203 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480\": container with ID starting with 2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480 not found: ID does not exist" containerID="2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480" Jul 28 00:06:19.540271 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.540227 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480"} err="failed to get container status \"2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480\": rpc error: code = NotFound desc = could not find container \"2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480\": container with ID starting with 2ffe1de3d503d2413a4acb8f1b27fd98d932ed1b519e1a064e95105e3f796480 not found: ID does not exist" Jul 28 00:06:19.540271 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.540242 2555 scope.go:117] "RemoveContainer" containerID="4244851249fedb0b11284f12ec6e5af10c86fd8a1c5ef2c7ad32c532746f7459" Jul 28 00:06:19.552815 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.552795 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x"] Jul 28 00:06:19.556215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.556195 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a3da35c-7c69697c6c-2n79x"] Jul 28 00:06:19.567226 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.567207 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s"] Jul 28 00:06:19.569611 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.569589 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a3da35c-97848f856-9vq2s"] Jul 28 00:06:19.578689 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.578670 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt"] Jul 28 00:06:19.582269 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:19.582249 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a3da35c-58cb4dc7f8-9ghwt"] Jul 28 00:06:20.564397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:20.564360 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0835a438-de38-464f-b3cc-cccea46fcaad" path="/var/lib/kubelet/pods/0835a438-de38-464f-b3cc-cccea46fcaad/volumes" Jul 28 00:06:20.564785 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:20.564722 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb609ea-1c4d-477b-a772-32de0cb8372b" path="/var/lib/kubelet/pods/3cb609ea-1c4d-477b-a772-32de0cb8372b/volumes" Jul 28 00:06:20.565030 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:20.565018 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" path="/var/lib/kubelet/pods/e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56/volumes" Jul 28 00:06:22.075378 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:22.075336 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr"] Jul 28 00:06:22.077509 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:22.077471 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a3da35c-bzgrr"] Jul 28 00:06:22.080285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:22.080262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:22.565123 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:22.565079 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae560bf6-7023-4dd2-ba10-8b396cda4a36" path="/var/lib/kubelet/pods/ae560bf6-7023-4dd2-ba10-8b396cda4a36/volumes" Jul 28 00:06:23.280388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:23.280338 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:23.504579 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:23.504540 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:23.725148 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:23.725053 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:24.012882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:24.012849 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:24.580066 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:24.580025 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:26.447840 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.447800 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:26.547543 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.547509 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:26.636115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636084 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f"] Jul 28 00:06:26.636462 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636447 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" containerName="payload-processing" Jul 28 00:06:26.636462 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636462 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" containerName="payload-processing" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636477 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb609ea-1c4d-477b-a772-32de0cb8372b" containerName="payload-pre-processing" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636483 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb609ea-1c4d-477b-a772-32de0cb8372b" containerName="payload-pre-processing" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636490 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae560bf6-7023-4dd2-ba10-8b396cda4a36" containerName="revoke-keys" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636511 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae560bf6-7023-4dd2-ba10-8b396cda4a36" containerName="revoke-keys" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636524 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0835a438-de38-464f-b3cc-cccea46fcaad" containerName="maas-api" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636529 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0835a438-de38-464f-b3cc-cccea46fcaad" containerName="maas-api" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636590 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27efbd5-545e-4ef5-bad7-3ae6cb0ecf56" containerName="payload-processing" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636598 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae560bf6-7023-4dd2-ba10-8b396cda4a36" containerName="revoke-keys" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636605 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb609ea-1c4d-477b-a772-32de0cb8372b" containerName="payload-pre-processing" Jul 28 00:06:26.636608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.636610 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0835a438-de38-464f-b3cc-cccea46fcaad" containerName="maas-api" Jul 28 00:06:26.641251 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.641229 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.643627 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.643605 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5b6cfa42-serving-cert\"" Jul 28 00:06:26.648245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.648219 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f"] Jul 28 00:06:26.677135 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.677102 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:26.709847 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.709767 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef75021-1b38-4694-8413-844a4e8f283f-openshift-service-ca\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.709847 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.709826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.710021 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.709860 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkgw6\" (UniqueName: \"kubernetes.io/projected/6ef75021-1b38-4694-8413-844a4e8f283f-kube-api-access-wkgw6\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.810589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.810560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.810746 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.810599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkgw6\" (UniqueName: \"kubernetes.io/projected/6ef75021-1b38-4694-8413-844a4e8f283f-kube-api-access-wkgw6\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.810746 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.810680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef75021-1b38-4694-8413-844a4e8f283f-openshift-service-ca\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.810746 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:06:26.810705 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-serving-cert: secret "maas-api-e2e-derive-5b6cfa42-serving-cert" not found Jul 28 00:06:26.810878 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:06:26.810779 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls podName:6ef75021-1b38-4694-8413-844a4e8f283f nodeName:}" failed. No retries permitted until 2026-07-28 00:06:27.310760606 +0000 UTC m=+1787.234969466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls") pod "maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" (UID: "6ef75021-1b38-4694-8413-844a4e8f283f") : secret "maas-api-e2e-derive-5b6cfa42-serving-cert" not found Jul 28 00:06:26.811339 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.811321 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef75021-1b38-4694-8413-844a4e8f283f-openshift-service-ca\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:26.818846 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:26.818828 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkgw6\" (UniqueName: \"kubernetes.io/projected/6ef75021-1b38-4694-8413-844a4e8f283f-kube-api-access-wkgw6\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:27.032830 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.032793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:27.310854 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.310778 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq"] Jul 28 00:06:27.313196 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.313169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:27.314413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.314397 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.315582 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.315562 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls\") pod \"maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:27.316660 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.316639 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5b6cfa42-dockercfg-bxkdc\"" Jul 28 00:06:27.316777 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.316760 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5b6cfa42\"" Jul 28 00:06:27.325136 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.325112 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq"] Jul 28 00:06:27.347470 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.347446 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq"] Jul 28 00:06:27.351228 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.351205 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.356721 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.356701 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq"] Jul 28 00:06:27.414257 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.414229 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f8b0b-0494-4a78-a9de-891851cf1040-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.414411 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.414265 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhl2d\" (UniqueName: \"kubernetes.io/projected/647f8b0b-0494-4a78-a9de-891851cf1040-kube-api-access-rhl2d\") pod \"payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.515620 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.515580 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f8b0b-0494-4a78-a9de-891851cf1040-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.515620 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.515620 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhl2d\" (UniqueName: \"kubernetes.io/projected/647f8b0b-0494-4a78-a9de-891851cf1040-kube-api-access-rhl2d\") pod \"payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.516039 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.515650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfl28\" (UniqueName: \"kubernetes.io/projected/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-kube-api-access-pfl28\") pod \"payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.516039 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.515683 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-plugins-config-volume\") pod \"payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.516283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.516263 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f8b0b-0494-4a78-a9de-891851cf1040-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.527920 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.527895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhl2d\" (UniqueName: \"kubernetes.io/projected/647f8b0b-0494-4a78-a9de-891851cf1040-kube-api-access-rhl2d\") pod \"payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.557694 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.557655 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:27.617052 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.617022 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfl28\" (UniqueName: \"kubernetes.io/projected/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-kube-api-access-pfl28\") pod \"payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.617199 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.617063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-plugins-config-volume\") pod \"payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.617833 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.617807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-plugins-config-volume\") pod \"payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.630887 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.630860 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfl28\" (UniqueName: \"kubernetes.io/projected/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-kube-api-access-pfl28\") pod \"payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.632590 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.632572 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:27.662470 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.662433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:27.710152 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.710127 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f"] Jul 28 00:06:27.713821 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:06:27.713780 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef75021_1b38_4694_8413_844a4e8f283f.slice/crio-9bc6917d80ba3d49a5f3724d9ec3a3249df9f60b8683000941cb48c38ae443f0 WatchSource:0}: Error finding container 9bc6917d80ba3d49a5f3724d9ec3a3249df9f60b8683000941cb48c38ae443f0: Status 404 returned error can't find the container with id 9bc6917d80ba3d49a5f3724d9ec3a3249df9f60b8683000941cb48c38ae443f0 Jul 28 00:06:27.791423 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.791396 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq"] Jul 28 00:06:27.794009 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:06:27.793977 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647f8b0b_0494_4a78_a9de_891851cf1040.slice/crio-e1d1531b9e13e246237556ee7a8192c19f154bcdfded5b0894cc259bc00c0322 WatchSource:0}: Error finding container e1d1531b9e13e246237556ee7a8192c19f154bcdfded5b0894cc259bc00c0322: Status 404 returned error can't find the container with id e1d1531b9e13e246237556ee7a8192c19f154bcdfded5b0894cc259bc00c0322 Jul 28 00:06:27.918173 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:27.918140 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:28.018655 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.018626 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:28.029838 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.029805 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq"] Jul 28 00:06:28.032834 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:06:28.032795 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f8f3e2_676b_4159_98ed_aa1fd2b4eb49.slice/crio-45694ad89e3d7b68b02b3710db6f824a35009e2a061467b1b6218e9a6ccfa0fb WatchSource:0}: Error finding container 45694ad89e3d7b68b02b3710db6f824a35009e2a061467b1b6218e9a6ccfa0fb: Status 404 returned error can't find the container with id 45694ad89e3d7b68b02b3710db6f824a35009e2a061467b1b6218e9a6ccfa0fb Jul 28 00:06:28.570850 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.570817 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" event={"ID":"647f8b0b-0494-4a78-a9de-891851cf1040","Type":"ContainerStarted","Data":"b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624"} Jul 28 00:06:28.571330 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.570869 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" event={"ID":"647f8b0b-0494-4a78-a9de-891851cf1040","Type":"ContainerStarted","Data":"e1d1531b9e13e246237556ee7a8192c19f154bcdfded5b0894cc259bc00c0322"} Jul 28 00:06:28.571330 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.570991 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:28.572267 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.572243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" event={"ID":"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49","Type":"ContainerStarted","Data":"74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42"} Jul 28 00:06:28.572405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.572292 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:28.572405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.572316 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" event={"ID":"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49","Type":"ContainerStarted","Data":"45694ad89e3d7b68b02b3710db6f824a35009e2a061467b1b6218e9a6ccfa0fb"} Jul 28 00:06:28.573822 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.573794 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" event={"ID":"6ef75021-1b38-4694-8413-844a4e8f283f","Type":"ContainerStarted","Data":"9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93"} Jul 28 00:06:28.574118 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.573830 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" event={"ID":"6ef75021-1b38-4694-8413-844a4e8f283f","Type":"ContainerStarted","Data":"9bc6917d80ba3d49a5f3724d9ec3a3249df9f60b8683000941cb48c38ae443f0"} Jul 28 00:06:28.574118 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.573852 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:28.584412 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.584332 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" podStartSLOduration=1.584303376 podStartE2EDuration="1.584303376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:06:28.583941407 +0000 UTC m=+1788.508150289" watchObservedRunningTime="2026-07-28 00:06:28.584303376 +0000 UTC m=+1788.508512255" Jul 28 00:06:28.599336 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.599292 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" podStartSLOduration=1.5992810020000001 podStartE2EDuration="1.599281002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:06:28.597791893 +0000 UTC m=+1788.522000771" watchObservedRunningTime="2026-07-28 00:06:28.599281002 +0000 UTC m=+1788.523489857" Jul 28 00:06:28.614399 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:28.614355 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" podStartSLOduration=2.255145613 podStartE2EDuration="2.614341826s" totalImagesPullingTime="359.196213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:06:28.613090378 +0000 UTC m=+1788.537299256" watchObservedRunningTime="2026-07-28 00:06:28.614341826 +0000 UTC m=+1788.538550704" Jul 28 00:06:34.585133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:34.585057 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:06:39.580670 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:39.580638 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:06:39.581073 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:39.580707 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:06:51.264515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.264341 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7"] Jul 28 00:06:51.270110 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.270075 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:51.271993 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.271963 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7"] Jul 28 00:06:51.273045 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.272250 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:06:51.323299 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.323263 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24343596-f497-43cb-b3ae-6011e85c7ecc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7\" (UID: \"24343596-f497-43cb-b3ae-6011e85c7ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:51.424679 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.424639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24343596-f497-43cb-b3ae-6011e85c7ecc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7\" (UID: \"24343596-f497-43cb-b3ae-6011e85c7ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:51.425323 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.425301 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24343596-f497-43cb-b3ae-6011e85c7ecc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7\" (UID: \"24343596-f497-43cb-b3ae-6011e85c7ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:51.582783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.582688 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:51.721410 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:51.721387 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7"] Jul 28 00:06:51.723359 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:06:51.723327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24343596_f497_43cb_b3ae_6011e85c7ecc.slice/crio-721ef87cd64ffce5f840bbc5a2fa5d80966dabc105fb35826a3ce05c227e7e01 WatchSource:0}: Error finding container 721ef87cd64ffce5f840bbc5a2fa5d80966dabc105fb35826a3ce05c227e7e01: Status 404 returned error can't find the container with id 721ef87cd64ffce5f840bbc5a2fa5d80966dabc105fb35826a3ce05c227e7e01 Jul 28 00:06:52.692069 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:52.691974 2555 generic.go:356] "Generic (PLEG): container finished" podID="24343596-f497-43cb-b3ae-6011e85c7ecc" containerID="2334323ea29d4dbe9d311788e28a212e0b8a285b6349e3bece2e0eff902422df" exitCode=0 Jul 28 00:06:52.692069 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:52.692055 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" event={"ID":"24343596-f497-43cb-b3ae-6011e85c7ecc","Type":"ContainerDied","Data":"2334323ea29d4dbe9d311788e28a212e0b8a285b6349e3bece2e0eff902422df"} Jul 28 00:06:52.692465 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:52.692107 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" event={"ID":"24343596-f497-43cb-b3ae-6011e85c7ecc","Type":"ContainerStarted","Data":"721ef87cd64ffce5f840bbc5a2fa5d80966dabc105fb35826a3ce05c227e7e01"} Jul 28 00:06:53.833625 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:53.833599 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:53.948521 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:53.948419 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24343596-f497-43cb-b3ae-6011e85c7ecc-maas-api-service-ca\") pod \"24343596-f497-43cb-b3ae-6011e85c7ecc\" (UID: \"24343596-f497-43cb-b3ae-6011e85c7ecc\") " Jul 28 00:06:53.948805 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:53.948786 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24343596-f497-43cb-b3ae-6011e85c7ecc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24343596-f497-43cb-b3ae-6011e85c7ecc" (UID: "24343596-f497-43cb-b3ae-6011e85c7ecc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:06:54.049859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:54.049827 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24343596-f497-43cb-b3ae-6011e85c7ecc-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:06:54.703437 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:54.703405 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" Jul 28 00:06:54.703656 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:54.703440 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7" event={"ID":"24343596-f497-43cb-b3ae-6011e85c7ecc","Type":"ContainerDied","Data":"721ef87cd64ffce5f840bbc5a2fa5d80966dabc105fb35826a3ce05c227e7e01"} Jul 28 00:06:54.703656 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:06:54.703490 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721ef87cd64ffce5f840bbc5a2fa5d80966dabc105fb35826a3ce05c227e7e01" Jul 28 00:07:01.652320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.652282 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f"] Jul 28 00:07:01.653187 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.653153 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" podUID="6ef75021-1b38-4694-8413-844a4e8f283f" containerName="maas-api" containerID="cri-o://9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93" gracePeriod=30 Jul 28 00:07:01.674147 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.674084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:01.720592 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.720556 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq"] Jul 28 00:07:01.720794 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.720772 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" podUID="c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" containerName="payload-processing" containerID="cri-o://74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42" gracePeriod=30 Jul 28 00:07:01.727310 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.726415 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq"] Jul 28 00:07:01.727310 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.726657 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" podUID="647f8b0b-0494-4a78-a9de-891851cf1040" containerName="payload-pre-processing" containerID="cri-o://b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624" gracePeriod=30 Jul 28 00:07:01.756173 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.756141 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:01.812390 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:01.812344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:02.005713 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.005687 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:07:02.008188 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.008165 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:07:02.031423 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.031396 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:07:02.119820 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119787 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef75021-1b38-4694-8413-844a4e8f283f-openshift-service-ca\") pod \"6ef75021-1b38-4694-8413-844a4e8f283f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " Jul 28 00:07:02.119820 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119820 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfl28\" (UniqueName: \"kubernetes.io/projected/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-kube-api-access-pfl28\") pod \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " Jul 28 00:07:02.120047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119840 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f8b0b-0494-4a78-a9de-891851cf1040-plugins-config-volume\") pod \"647f8b0b-0494-4a78-a9de-891851cf1040\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " Jul 28 00:07:02.120047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119894 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-plugins-config-volume\") pod \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\" (UID: \"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49\") " Jul 28 00:07:02.120047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119912 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls\") pod \"6ef75021-1b38-4694-8413-844a4e8f283f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " Jul 28 00:07:02.120047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119949 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhl2d\" (UniqueName: \"kubernetes.io/projected/647f8b0b-0494-4a78-a9de-891851cf1040-kube-api-access-rhl2d\") pod \"647f8b0b-0494-4a78-a9de-891851cf1040\" (UID: \"647f8b0b-0494-4a78-a9de-891851cf1040\") " Jul 28 00:07:02.120047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.119997 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkgw6\" (UniqueName: \"kubernetes.io/projected/6ef75021-1b38-4694-8413-844a4e8f283f-kube-api-access-wkgw6\") pod \"6ef75021-1b38-4694-8413-844a4e8f283f\" (UID: \"6ef75021-1b38-4694-8413-844a4e8f283f\") " Jul 28 00:07:02.120309 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.120184 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef75021-1b38-4694-8413-844a4e8f283f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ef75021-1b38-4694-8413-844a4e8f283f" (UID: "6ef75021-1b38-4694-8413-844a4e8f283f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:07:02.120309 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.120291 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647f8b0b-0494-4a78-a9de-891851cf1040-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "647f8b0b-0494-4a78-a9de-891851cf1040" (UID: "647f8b0b-0494-4a78-a9de-891851cf1040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:07:02.120309 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.120302 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef75021-1b38-4694-8413-844a4e8f283f-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.120532 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.120306 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" (UID: "c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:07:02.122031 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.121996 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ef75021-1b38-4694-8413-844a4e8f283f" (UID: "6ef75021-1b38-4694-8413-844a4e8f283f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:07:02.122148 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.122075 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-kube-api-access-pfl28" (OuterVolumeSpecName: "kube-api-access-pfl28") pod "c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" (UID: "c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49"). InnerVolumeSpecName "kube-api-access-pfl28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:07:02.122148 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.122086 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647f8b0b-0494-4a78-a9de-891851cf1040-kube-api-access-rhl2d" (OuterVolumeSpecName: "kube-api-access-rhl2d") pod "647f8b0b-0494-4a78-a9de-891851cf1040" (UID: "647f8b0b-0494-4a78-a9de-891851cf1040"). InnerVolumeSpecName "kube-api-access-rhl2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:07:02.122250 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.122182 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef75021-1b38-4694-8413-844a4e8f283f-kube-api-access-wkgw6" (OuterVolumeSpecName: "kube-api-access-wkgw6") pod "6ef75021-1b38-4694-8413-844a4e8f283f" (UID: "6ef75021-1b38-4694-8413-844a4e8f283f"). InnerVolumeSpecName "kube-api-access-wkgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:07:02.221706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.221613 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfl28\" (UniqueName: \"kubernetes.io/projected/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-kube-api-access-pfl28\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.221706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.221645 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f8b0b-0494-4a78-a9de-891851cf1040-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.221706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.221654 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.221706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.221668 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef75021-1b38-4694-8413-844a4e8f283f-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.221706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.221683 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhl2d\" (UniqueName: \"kubernetes.io/projected/647f8b0b-0494-4a78-a9de-891851cf1040-kube-api-access-rhl2d\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.221706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.221696 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkgw6\" (UniqueName: \"kubernetes.io/projected/6ef75021-1b38-4694-8413-844a4e8f283f-kube-api-access-wkgw6\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:07:02.381019 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.380978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:02.749264 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.749226 2555 generic.go:356] "Generic (PLEG): container finished" podID="647f8b0b-0494-4a78-a9de-891851cf1040" containerID="b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624" exitCode=0 Jul 28 00:07:02.749754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.749277 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" Jul 28 00:07:02.749754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.749304 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" event={"ID":"647f8b0b-0494-4a78-a9de-891851cf1040","Type":"ContainerDied","Data":"b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624"} Jul 28 00:07:02.749754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.749388 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq" event={"ID":"647f8b0b-0494-4a78-a9de-891851cf1040","Type":"ContainerDied","Data":"e1d1531b9e13e246237556ee7a8192c19f154bcdfded5b0894cc259bc00c0322"} Jul 28 00:07:02.749754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.749420 2555 scope.go:117] "RemoveContainer" containerID="b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624" Jul 28 00:07:02.750806 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.750750 2555 generic.go:356] "Generic (PLEG): container finished" podID="c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" containerID="74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42" exitCode=0 Jul 28 00:07:02.750806 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.750785 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" event={"ID":"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49","Type":"ContainerDied","Data":"74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42"} Jul 28 00:07:02.750941 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.750818 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" Jul 28 00:07:02.750941 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.750836 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq" event={"ID":"c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49","Type":"ContainerDied","Data":"45694ad89e3d7b68b02b3710db6f824a35009e2a061467b1b6218e9a6ccfa0fb"} Jul 28 00:07:02.752409 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.752378 2555 generic.go:356] "Generic (PLEG): container finished" podID="6ef75021-1b38-4694-8413-844a4e8f283f" containerID="9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93" exitCode=0 Jul 28 00:07:02.752547 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.752408 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" event={"ID":"6ef75021-1b38-4694-8413-844a4e8f283f","Type":"ContainerDied","Data":"9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93"} Jul 28 00:07:02.752547 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.752445 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" Jul 28 00:07:02.752547 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.752447 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f" event={"ID":"6ef75021-1b38-4694-8413-844a4e8f283f","Type":"ContainerDied","Data":"9bc6917d80ba3d49a5f3724d9ec3a3249df9f60b8683000941cb48c38ae443f0"} Jul 28 00:07:02.759207 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.759186 2555 scope.go:117] "RemoveContainer" containerID="b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624" Jul 28 00:07:02.759445 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:07:02.759421 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624\": container with ID starting with b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624 not found: ID does not exist" containerID="b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624" Jul 28 00:07:02.759544 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.759456 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624"} err="failed to get container status \"b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624\": rpc error: code = NotFound desc = could not find container \"b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624\": container with ID starting with b969b4a8175aae26f30c48777b121d6522b264a8c2c844c1a57e0da6d3788624 not found: ID does not exist" Jul 28 00:07:02.759544 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.759478 2555 scope.go:117] "RemoveContainer" containerID="74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42" Jul 28 00:07:02.767104 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.767087 2555 scope.go:117] "RemoveContainer" containerID="74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42" Jul 28 00:07:02.767360 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:07:02.767342 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42\": container with ID starting with 74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42 not found: ID does not exist" containerID="74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42" Jul 28 00:07:02.767408 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.767367 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42"} err="failed to get container status \"74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42\": rpc error: code = NotFound desc = could not find container \"74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42\": container with ID starting with 74e05bba5c3978d4b0cc242238f40fa54a7c7d89438cbb8b2c3b73a72834af42 not found: ID does not exist" Jul 28 00:07:02.767408 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.767388 2555 scope.go:117] "RemoveContainer" containerID="9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93" Jul 28 00:07:02.774538 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.774520 2555 scope.go:117] "RemoveContainer" containerID="9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93" Jul 28 00:07:02.774769 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:07:02.774754 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93\": container with ID starting with 9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93 not found: ID does not exist" containerID="9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93" Jul 28 00:07:02.774817 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.774773 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93"} err="failed to get container status \"9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93\": rpc error: code = NotFound desc = could not find container \"9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93\": container with ID starting with 9ab4f46c22228eeb3df4999558b919ffb5f005dc36f7c983b3943b4c380a7e93 not found: ID does not exist" Jul 28 00:07:02.778538 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.778516 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq"] Jul 28 00:07:02.784115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.784097 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5b6cfa42-68fb7bf66-nxdgq"] Jul 28 00:07:02.802482 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.802455 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f"] Jul 28 00:07:02.808673 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.808647 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5b6cfa42-5c9676fbbd-ds28f"] Jul 28 00:07:02.820369 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.820345 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq"] Jul 28 00:07:02.824162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:02.824142 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5b6cfa42-67db78b8-8ctcq"] Jul 28 00:07:04.565605 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:04.565569 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647f8b0b-0494-4a78-a9de-891851cf1040" path="/var/lib/kubelet/pods/647f8b0b-0494-4a78-a9de-891851cf1040/volumes" Jul 28 00:07:04.565985 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:04.565971 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef75021-1b38-4694-8413-844a4e8f283f" path="/var/lib/kubelet/pods/6ef75021-1b38-4694-8413-844a4e8f283f/volumes" Jul 28 00:07:04.566283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:04.566272 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" path="/var/lib/kubelet/pods/c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49/volumes" Jul 28 00:07:06.333892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:06.333858 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7"] Jul 28 00:07:06.335633 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:06.335607 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b6cfa42-k7fj7"] Jul 28 00:07:06.336301 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:06.336247 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:06.496410 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:06.496371 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:06.564605 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:06.564571 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24343596-f497-43cb-b3ae-6011e85c7ecc" path="/var/lib/kubelet/pods/24343596-f497-43cb-b3ae-6011e85c7ecc/volumes" Jul 28 00:07:06.740882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:06.740849 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:07.234185 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:07.234090 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:07.337272 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:07.337237 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:07.433400 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:07.433366 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:08.840100 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:08.840062 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:15.068507 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:15.068472 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:15.175832 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:15.175793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:15.276405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:15.276363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:15.375180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:15.375092 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:15.808214 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:15.808175 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:41.764760 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:41.764719 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:42.074592 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:42.074485 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:42.128218 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:42.128179 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:48.254569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:48.254535 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:48.352313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:48.352257 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:48.451989 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:07:48.451930 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:04.350291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:04.350194 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:09.891743 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:09.891707 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:10.014444 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:10.014410 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:10.134229 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:10.134193 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:20.662105 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:20.662070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:21.513083 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:21.513039 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:27.852795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:27.852755 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:27.963432 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:27.963402 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:28.052993 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:28.052955 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:28.153382 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:28.153288 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:33.318852 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:33.318811 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:33.418444 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:33.418385 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:33.518937 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:33.518893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:35.350820 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:35.350783 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:41.162772 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:41.162734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:41.221174 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:41.221142 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:41.867647 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:41.867610 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:47.594935 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:47.594889 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:47.695431 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:47.695392 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:47.850327 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:47.850235 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:48.240775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:48.240713 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:48.501082 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:48.501004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:51.164443 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.164408 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:51.251070 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251038 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945"] Jul 28 00:08:51.251586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251567 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24343596-f497-43cb-b3ae-6011e85c7ecc" containerName="revoke-keys" Jul 28 00:08:51.251586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251586 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24343596-f497-43cb-b3ae-6011e85c7ecc" containerName="revoke-keys" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251602 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" containerName="payload-processing" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251609 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" containerName="payload-processing" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251623 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647f8b0b-0494-4a78-a9de-891851cf1040" containerName="payload-pre-processing" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251629 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="647f8b0b-0494-4a78-a9de-891851cf1040" containerName="payload-pre-processing" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251638 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef75021-1b38-4694-8413-844a4e8f283f" containerName="maas-api" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251645 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef75021-1b38-4694-8413-844a4e8f283f" containerName="maas-api" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251705 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef75021-1b38-4694-8413-844a4e8f283f" containerName="maas-api" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251712 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f8f3e2-676b-4159-98ed-aa1fd2b4eb49" containerName="payload-processing" Jul 28 00:08:51.251714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251719 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="647f8b0b-0494-4a78-a9de-891851cf1040" containerName="payload-pre-processing" Jul 28 00:08:51.252227 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.251725 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24343596-f497-43cb-b3ae-6011e85c7ecc" containerName="revoke-keys" Jul 28 00:08:51.254222 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.254206 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.259274 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.259248 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8c8f285e-serving-cert\"" Jul 28 00:08:51.269881 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.269860 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945"] Jul 28 00:08:51.270195 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.270173 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:51.360517 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.360450 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwjf\" (UniqueName: \"kubernetes.io/projected/43c94762-87c2-44ff-8efc-39cc6fb52581-kube-api-access-cnwjf\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.360707 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.360583 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c94762-87c2-44ff-8efc-39cc6fb52581-maas-api-tls\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.360707 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.360684 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c94762-87c2-44ff-8efc-39cc6fb52581-openshift-service-ca\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.381903 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.381867 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:51.461569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.461459 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c94762-87c2-44ff-8efc-39cc6fb52581-openshift-service-ca\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.461569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.461552 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwjf\" (UniqueName: \"kubernetes.io/projected/43c94762-87c2-44ff-8efc-39cc6fb52581-kube-api-access-cnwjf\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.461822 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.461607 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c94762-87c2-44ff-8efc-39cc6fb52581-maas-api-tls\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.462179 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.462154 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c94762-87c2-44ff-8efc-39cc6fb52581-openshift-service-ca\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.463995 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.463973 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c94762-87c2-44ff-8efc-39cc6fb52581-maas-api-tls\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.478870 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.478838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwjf\" (UniqueName: \"kubernetes.io/projected/43c94762-87c2-44ff-8efc-39cc6fb52581-kube-api-access-cnwjf\") pod \"maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.511612 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.511573 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:51.565226 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.565189 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:51.704031 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.703998 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945"] Jul 28 00:08:51.707294 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:08:51.707267 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c94762_87c2_44ff_8efc_39cc6fb52581.slice/crio-0af869418864a8e8f1295d56911dd31d8d85c44267a89eb3a808de9e9f07b2c5 WatchSource:0}: Error finding container 0af869418864a8e8f1295d56911dd31d8d85c44267a89eb3a808de9e9f07b2c5: Status 404 returned error can't find the container with id 0af869418864a8e8f1295d56911dd31d8d85c44267a89eb3a808de9e9f07b2c5 Jul 28 00:08:51.918894 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:51.918864 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:52.014744 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.014701 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:52.022772 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.022747 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk"] Jul 28 00:08:52.026549 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.026533 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.028901 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.028880 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8c8f285e-dockercfg-dxsch\"" Jul 28 00:08:52.029026 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.028923 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8c8f285e\"" Jul 28 00:08:52.035709 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.035687 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk"] Jul 28 00:08:52.052928 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.052901 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq"] Jul 28 00:08:52.056616 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.056599 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.063954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.063931 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq"] Jul 28 00:08:52.167806 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.167777 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhkzt\" (UniqueName: \"kubernetes.io/projected/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-kube-api-access-xhkzt\") pod \"payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.168121 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.167821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9063e-a023-4406-af0f-3685f0d24653-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.168121 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.167844 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hs8\" (UniqueName: \"kubernetes.io/projected/57b9063e-a023-4406-af0f-3685f0d24653-kube-api-access-b5hs8\") pod \"payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.168121 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.167912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-plugins-config-volume\") pod \"payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.269045 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.269007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhkzt\" (UniqueName: \"kubernetes.io/projected/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-kube-api-access-xhkzt\") pod \"payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.269045 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.269053 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9063e-a023-4406-af0f-3685f0d24653-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.269298 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.269168 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hs8\" (UniqueName: \"kubernetes.io/projected/57b9063e-a023-4406-af0f-3685f0d24653-kube-api-access-b5hs8\") pod \"payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.269298 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.269222 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-plugins-config-volume\") pod \"payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.269868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.269851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9063e-a023-4406-af0f-3685f0d24653-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.269933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.269913 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-plugins-config-volume\") pod \"payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.276400 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.276365 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhkzt\" (UniqueName: \"kubernetes.io/projected/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-kube-api-access-xhkzt\") pod \"payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.276400 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.276393 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hs8\" (UniqueName: \"kubernetes.io/projected/57b9063e-a023-4406-af0f-3685f0d24653-kube-api-access-b5hs8\") pod \"payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.280775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.280274 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" event={"ID":"43c94762-87c2-44ff-8efc-39cc6fb52581","Type":"ContainerStarted","Data":"b33512c40ea52d340d7f41e14d1e6ce4631ea78c21e106a32c416f1ef1e0875e"} Jul 28 00:08:52.280775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.280462 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:08:52.280775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.280541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" event={"ID":"43c94762-87c2-44ff-8efc-39cc6fb52581","Type":"ContainerStarted","Data":"0af869418864a8e8f1295d56911dd31d8d85c44267a89eb3a808de9e9f07b2c5"} Jul 28 00:08:52.298753 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.298699 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" podStartSLOduration=0.885429403 podStartE2EDuration="1.298680321s" totalImagesPullingTime="413.250918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:52.296796201 +0000 UTC m=+1932.221005092" watchObservedRunningTime="2026-07-28 00:08:52.298680321 +0000 UTC m=+1932.222889201" Jul 28 00:08:52.338126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.338099 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:52.367276 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.367250 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:52.516656 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.516409 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq"] Jul 28 00:08:52.518824 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:08:52.518792 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a52d57b_4d94_4a9a_b7fc_cd1013ac5aeb.slice/crio-0e1288ecb79bef963c349d44adac739d06e6fd80a3092f5d2d7ac87b915b2ffd WatchSource:0}: Error finding container 0e1288ecb79bef963c349d44adac739d06e6fd80a3092f5d2d7ac87b915b2ffd: Status 404 returned error can't find the container with id 0e1288ecb79bef963c349d44adac739d06e6fd80a3092f5d2d7ac87b915b2ffd Jul 28 00:08:52.691208 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:52.691179 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk"] Jul 28 00:08:52.693307 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:08:52.693279 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b9063e_a023_4406_af0f_3685f0d24653.slice/crio-e715d6936ebf3fbf1b547d17726a5148b17a1779962ff343307f95a1a1f823f9 WatchSource:0}: Error finding container e715d6936ebf3fbf1b547d17726a5148b17a1779962ff343307f95a1a1f823f9: Status 404 returned error can't find the container with id e715d6936ebf3fbf1b547d17726a5148b17a1779962ff343307f95a1a1f823f9 Jul 28 00:08:53.288427 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.288391 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" event={"ID":"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb","Type":"ContainerStarted","Data":"21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc"} Jul 28 00:08:53.288901 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.288443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" event={"ID":"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb","Type":"ContainerStarted","Data":"0e1288ecb79bef963c349d44adac739d06e6fd80a3092f5d2d7ac87b915b2ffd"} Jul 28 00:08:53.288901 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.288482 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:08:53.290294 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.290267 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" event={"ID":"57b9063e-a023-4406-af0f-3685f0d24653","Type":"ContainerStarted","Data":"33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a"} Jul 28 00:08:53.290447 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.290317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" event={"ID":"57b9063e-a023-4406-af0f-3685f0d24653","Type":"ContainerStarted","Data":"e715d6936ebf3fbf1b547d17726a5148b17a1779962ff343307f95a1a1f823f9"} Jul 28 00:08:53.290447 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.290424 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:08:53.309874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.309829 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" podStartSLOduration=1.309814117 podStartE2EDuration="1.309814117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:53.308565748 +0000 UTC m=+1933.232774649" watchObservedRunningTime="2026-07-28 00:08:53.309814117 +0000 UTC m=+1933.234023393" Jul 28 00:08:53.327099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:53.327039 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" podStartSLOduration=1.326939133 podStartE2EDuration="1.326939133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:53.325896258 +0000 UTC m=+1933.250105139" watchObservedRunningTime="2026-07-28 00:08:53.326939133 +0000 UTC m=+1933.251148003" Jul 28 00:08:55.745266 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:55.745231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:56.407482 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:56.407440 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:56.506976 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:56.506942 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:56.857375 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:56.857342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:58.295670 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:08:58.295641 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:09:02.473668 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.473622 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:02.573575 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.573532 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:02.849024 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.848988 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq"] Jul 28 00:09:02.854628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.854607 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:02.856962 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.856936 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:09:02.859961 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.859624 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq"] Jul 28 00:09:02.865586 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.865557 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed40fc14-6e19-4f69-8954-65a3b453427e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq\" (UID: \"ed40fc14-6e19-4f69-8954-65a3b453427e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:02.966679 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.966641 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed40fc14-6e19-4f69-8954-65a3b453427e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq\" (UID: \"ed40fc14-6e19-4f69-8954-65a3b453427e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:02.967337 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:02.967314 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed40fc14-6e19-4f69-8954-65a3b453427e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq\" (UID: \"ed40fc14-6e19-4f69-8954-65a3b453427e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:03.167369 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:03.167281 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:03.307683 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:03.307596 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq"] Jul 28 00:09:03.310588 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:09:03.310560 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded40fc14_6e19_4f69_8954_65a3b453427e.slice/crio-7b67eee5e063c3e465c0b7f7252601ad786d8903204908953ef511e8e2d80db0 WatchSource:0}: Error finding container 7b67eee5e063c3e465c0b7f7252601ad786d8903204908953ef511e8e2d80db0: Status 404 returned error can't find the container with id 7b67eee5e063c3e465c0b7f7252601ad786d8903204908953ef511e8e2d80db0 Jul 28 00:09:03.340427 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:03.340394 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" event={"ID":"ed40fc14-6e19-4f69-8954-65a3b453427e","Type":"ContainerStarted","Data":"7b67eee5e063c3e465c0b7f7252601ad786d8903204908953ef511e8e2d80db0"} Jul 28 00:09:04.297878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:04.297845 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:09:04.298255 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:04.297997 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:09:04.347684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:04.347576 2555 generic.go:356] "Generic (PLEG): container finished" podID="ed40fc14-6e19-4f69-8954-65a3b453427e" containerID="a08508c17c682ecaa6d7735f2517a6788da4a09559051d1c3f3e27ffc2a39907" exitCode=0 Jul 28 00:09:04.347684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:04.347631 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" event={"ID":"ed40fc14-6e19-4f69-8954-65a3b453427e","Type":"ContainerDied","Data":"a08508c17c682ecaa6d7735f2517a6788da4a09559051d1c3f3e27ffc2a39907"} Jul 28 00:09:05.484930 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:05.484906 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:05.586328 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:05.586294 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed40fc14-6e19-4f69-8954-65a3b453427e-maas-api-service-ca\") pod \"ed40fc14-6e19-4f69-8954-65a3b453427e\" (UID: \"ed40fc14-6e19-4f69-8954-65a3b453427e\") " Jul 28 00:09:05.586723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:05.586700 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed40fc14-6e19-4f69-8954-65a3b453427e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed40fc14-6e19-4f69-8954-65a3b453427e" (UID: "ed40fc14-6e19-4f69-8954-65a3b453427e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:09:05.687373 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:05.687294 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed40fc14-6e19-4f69-8954-65a3b453427e-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:06.358027 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:06.357994 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" Jul 28 00:09:06.358027 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:06.358009 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq" event={"ID":"ed40fc14-6e19-4f69-8954-65a3b453427e","Type":"ContainerDied","Data":"7b67eee5e063c3e465c0b7f7252601ad786d8903204908953ef511e8e2d80db0"} Jul 28 00:09:06.358247 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:06.358061 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b67eee5e063c3e465c0b7f7252601ad786d8903204908953ef511e8e2d80db0" Jul 28 00:09:13.226642 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.226158 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945"] Jul 28 00:09:13.226642 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.226392 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" podUID="43c94762-87c2-44ff-8efc-39cc6fb52581" containerName="maas-api" containerID="cri-o://b33512c40ea52d340d7f41e14d1e6ce4631ea78c21e106a32c416f1ef1e0875e" gracePeriod=30 Jul 28 00:09:13.255326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.255280 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:13.282668 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.282616 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq"] Jul 28 00:09:13.282903 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.282860 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" podUID="9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" containerName="payload-processing" containerID="cri-o://21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc" gracePeriod=30 Jul 28 00:09:13.288734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.288709 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk"] Jul 28 00:09:13.288938 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.288912 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" podUID="57b9063e-a023-4406-af0f-3685f0d24653" containerName="payload-pre-processing" containerID="cri-o://33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a" gracePeriod=30 Jul 28 00:09:13.291580 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.291458 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" podUID="43c94762-87c2-44ff-8efc-39cc6fb52581" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.66:8443/health\": dial tcp 10.133.0.66:8443: connect: connection refused" Jul 28 00:09:13.355316 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.355285 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:13.394714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.394681 2555 generic.go:356] "Generic (PLEG): container finished" podID="43c94762-87c2-44ff-8efc-39cc6fb52581" containerID="b33512c40ea52d340d7f41e14d1e6ce4631ea78c21e106a32c416f1ef1e0875e" exitCode=0 Jul 28 00:09:13.394889 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.394755 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" event={"ID":"43c94762-87c2-44ff-8efc-39cc6fb52581","Type":"ContainerDied","Data":"b33512c40ea52d340d7f41e14d1e6ce4631ea78c21e106a32c416f1ef1e0875e"} Jul 28 00:09:13.454163 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.454127 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:13.574633 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.574606 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:09:13.577364 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.577346 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:09:13.600048 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.600017 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:09:13.656129 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656094 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnwjf\" (UniqueName: \"kubernetes.io/projected/43c94762-87c2-44ff-8efc-39cc6fb52581-kube-api-access-cnwjf\") pod \"43c94762-87c2-44ff-8efc-39cc6fb52581\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " Jul 28 00:09:13.656129 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656139 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c94762-87c2-44ff-8efc-39cc6fb52581-openshift-service-ca\") pod \"43c94762-87c2-44ff-8efc-39cc6fb52581\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " Jul 28 00:09:13.656406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656175 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c94762-87c2-44ff-8efc-39cc6fb52581-maas-api-tls\") pod \"43c94762-87c2-44ff-8efc-39cc6fb52581\" (UID: \"43c94762-87c2-44ff-8efc-39cc6fb52581\") " Jul 28 00:09:13.656406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656199 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhkzt\" (UniqueName: \"kubernetes.io/projected/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-kube-api-access-xhkzt\") pod \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " Jul 28 00:09:13.656406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656223 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5hs8\" (UniqueName: \"kubernetes.io/projected/57b9063e-a023-4406-af0f-3685f0d24653-kube-api-access-b5hs8\") pod \"57b9063e-a023-4406-af0f-3685f0d24653\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " Jul 28 00:09:13.656406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656242 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9063e-a023-4406-af0f-3685f0d24653-plugins-config-volume\") pod \"57b9063e-a023-4406-af0f-3685f0d24653\" (UID: \"57b9063e-a023-4406-af0f-3685f0d24653\") " Jul 28 00:09:13.656406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656265 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-plugins-config-volume\") pod \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\" (UID: \"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb\") " Jul 28 00:09:13.656709 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656588 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c94762-87c2-44ff-8efc-39cc6fb52581-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43c94762-87c2-44ff-8efc-39cc6fb52581" (UID: "43c94762-87c2-44ff-8efc-39cc6fb52581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:09:13.656861 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656834 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b9063e-a023-4406-af0f-3685f0d24653-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57b9063e-a023-4406-af0f-3685f0d24653" (UID: "57b9063e-a023-4406-af0f-3685f0d24653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:09:13.656928 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.656852 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" (UID: "9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:09:13.658464 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.658441 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c94762-87c2-44ff-8efc-39cc6fb52581-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43c94762-87c2-44ff-8efc-39cc6fb52581" (UID: "43c94762-87c2-44ff-8efc-39cc6fb52581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:09:13.658464 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.658446 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-kube-api-access-xhkzt" (OuterVolumeSpecName: "kube-api-access-xhkzt") pod "9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" (UID: "9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb"). InnerVolumeSpecName "kube-api-access-xhkzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:09:13.658739 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.658717 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b9063e-a023-4406-af0f-3685f0d24653-kube-api-access-b5hs8" (OuterVolumeSpecName: "kube-api-access-b5hs8") pod "57b9063e-a023-4406-af0f-3685f0d24653" (UID: "57b9063e-a023-4406-af0f-3685f0d24653"). InnerVolumeSpecName "kube-api-access-b5hs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:09:13.658851 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.658834 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c94762-87c2-44ff-8efc-39cc6fb52581-kube-api-access-cnwjf" (OuterVolumeSpecName: "kube-api-access-cnwjf") pod "43c94762-87c2-44ff-8efc-39cc6fb52581" (UID: "43c94762-87c2-44ff-8efc-39cc6fb52581"). InnerVolumeSpecName "kube-api-access-cnwjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:09:13.757146 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757117 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c94762-87c2-44ff-8efc-39cc6fb52581-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:13.757146 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757146 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhkzt\" (UniqueName: \"kubernetes.io/projected/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-kube-api-access-xhkzt\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:13.757334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757157 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hs8\" (UniqueName: \"kubernetes.io/projected/57b9063e-a023-4406-af0f-3685f0d24653-kube-api-access-b5hs8\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:13.757334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757167 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9063e-a023-4406-af0f-3685f0d24653-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:13.757334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757175 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:13.757334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757184 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwjf\" (UniqueName: \"kubernetes.io/projected/43c94762-87c2-44ff-8efc-39cc6fb52581-kube-api-access-cnwjf\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:13.757334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:13.757192 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c94762-87c2-44ff-8efc-39cc6fb52581-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:09:14.400594 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.400560 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" Jul 28 00:09:14.401056 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.400560 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945" event={"ID":"43c94762-87c2-44ff-8efc-39cc6fb52581","Type":"ContainerDied","Data":"0af869418864a8e8f1295d56911dd31d8d85c44267a89eb3a808de9e9f07b2c5"} Jul 28 00:09:14.401056 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.400730 2555 scope.go:117] "RemoveContainer" containerID="b33512c40ea52d340d7f41e14d1e6ce4631ea78c21e106a32c416f1ef1e0875e" Jul 28 00:09:14.402168 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.402071 2555 generic.go:356] "Generic (PLEG): container finished" podID="9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" containerID="21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc" exitCode=0 Jul 28 00:09:14.402168 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.402123 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" Jul 28 00:09:14.402168 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.402142 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" event={"ID":"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb","Type":"ContainerDied","Data":"21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc"} Jul 28 00:09:14.402333 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.402197 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq" event={"ID":"9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb","Type":"ContainerDied","Data":"0e1288ecb79bef963c349d44adac739d06e6fd80a3092f5d2d7ac87b915b2ffd"} Jul 28 00:09:14.403721 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.403697 2555 generic.go:356] "Generic (PLEG): container finished" podID="57b9063e-a023-4406-af0f-3685f0d24653" containerID="33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a" exitCode=0 Jul 28 00:09:14.403876 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.403860 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" event={"ID":"57b9063e-a023-4406-af0f-3685f0d24653","Type":"ContainerDied","Data":"33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a"} Jul 28 00:09:14.403952 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.403904 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" event={"ID":"57b9063e-a023-4406-af0f-3685f0d24653","Type":"ContainerDied","Data":"e715d6936ebf3fbf1b547d17726a5148b17a1779962ff343307f95a1a1f823f9"} Jul 28 00:09:14.403952 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.403932 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk" Jul 28 00:09:14.410729 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.410710 2555 scope.go:117] "RemoveContainer" containerID="21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc" Jul 28 00:09:14.420208 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.420188 2555 scope.go:117] "RemoveContainer" containerID="21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc" Jul 28 00:09:14.420467 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:09:14.420445 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc\": container with ID starting with 21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc not found: ID does not exist" containerID="21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc" Jul 28 00:09:14.420543 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.420475 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc"} err="failed to get container status \"21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc\": rpc error: code = NotFound desc = could not find container \"21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc\": container with ID starting with 21d7d69a0047dff9471acccc7f6cee87b2c551d25266984eb15e1477f9467fbc not found: ID does not exist" Jul 28 00:09:14.420543 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.420514 2555 scope.go:117] "RemoveContainer" containerID="33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a" Jul 28 00:09:14.426932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.426909 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945"] Jul 28 00:09:14.428927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.428909 2555 scope.go:117] "RemoveContainer" containerID="33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a" Jul 28 00:09:14.429190 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:09:14.429172 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a\": container with ID starting with 33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a not found: ID does not exist" containerID="33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a" Jul 28 00:09:14.429244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.429199 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a"} err="failed to get container status \"33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a\": rpc error: code = NotFound desc = could not find container \"33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a\": container with ID starting with 33e4df816680a771707a34960b3b2e352c4bbec8cab063870856a48f319f889a not found: ID does not exist" Jul 28 00:09:14.433121 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.433099 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c8f285e-565fc6d48b-rm945"] Jul 28 00:09:14.441397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.441372 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk"] Jul 28 00:09:14.443997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.443977 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c8f285e-585b9dbfc6-c6bsk"] Jul 28 00:09:14.451939 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.451916 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq"] Jul 28 00:09:14.455391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.455367 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c8f285e-648b5f544b-mj4lq"] Jul 28 00:09:14.565794 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.565711 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c94762-87c2-44ff-8efc-39cc6fb52581" path="/var/lib/kubelet/pods/43c94762-87c2-44ff-8efc-39cc6fb52581/volumes" Jul 28 00:09:14.566077 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.566064 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b9063e-a023-4406-af0f-3685f0d24653" path="/var/lib/kubelet/pods/57b9063e-a023-4406-af0f-3685f0d24653/volumes" Jul 28 00:09:14.566374 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:14.566363 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" path="/var/lib/kubelet/pods/9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb/volumes" Jul 28 00:09:17.966550 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:17.966509 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:17.975869 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:17.975837 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq"] Jul 28 00:09:17.979908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:17.979867 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c8f285e-4nvwq"] Jul 28 00:09:18.352200 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:18.352165 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:18.565251 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:18.565213 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed40fc14-6e19-4f69-8954-65a3b453427e" path="/var/lib/kubelet/pods/ed40fc14-6e19-4f69-8954-65a3b453427e/volumes" Jul 28 00:09:24.076639 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.076591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:24.176564 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.176527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:24.347717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.347619 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:24.586369 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.586336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:24.720919 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.720837 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:24.820650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.820603 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:24.961720 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:24.961677 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:30.774074 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:30.774041 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:30.878080 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:30.878045 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:30.978093 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:30.978046 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:31.115917 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:31.115817 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:36.824524 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:36.824419 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:36.921423 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:36.921389 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:43.914954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:43.914919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:44.241399 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:44.241368 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:44.341724 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:44.341678 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:47.015343 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.015301 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:47.109988 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.109935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:47.237556 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.237519 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5"] Jul 28 00:09:47.238082 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238065 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b9063e-a023-4406-af0f-3685f0d24653" containerName="payload-pre-processing" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238085 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b9063e-a023-4406-af0f-3685f0d24653" containerName="payload-pre-processing" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238109 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" containerName="payload-processing" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238118 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" containerName="payload-processing" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238136 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed40fc14-6e19-4f69-8954-65a3b453427e" containerName="revoke-keys" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238144 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed40fc14-6e19-4f69-8954-65a3b453427e" containerName="revoke-keys" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238161 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c94762-87c2-44ff-8efc-39cc6fb52581" containerName="maas-api" Jul 28 00:09:47.238175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238171 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c94762-87c2-44ff-8efc-39cc6fb52581" containerName="maas-api" Jul 28 00:09:47.238391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238253 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a52d57b-4d94-4a9a-b7fc-cd1013ac5aeb" containerName="payload-processing" Jul 28 00:09:47.238391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238266 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b9063e-a023-4406-af0f-3685f0d24653" containerName="payload-pre-processing" Jul 28 00:09:47.238391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238275 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed40fc14-6e19-4f69-8954-65a3b453427e" containerName="revoke-keys" Jul 28 00:09:47.238391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.238287 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c94762-87c2-44ff-8efc-39cc6fb52581" containerName="maas-api" Jul 28 00:09:47.241528 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.241506 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.243635 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.243616 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-352df9d8-serving-cert\"" Jul 28 00:09:47.250912 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.250889 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5"] Jul 28 00:09:47.273239 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.273158 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:47.347056 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.347020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada10ddd-01bd-4521-b32d-872a8908d58d-openshift-service-ca\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.347056 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.347056 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.347284 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.347079 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89c96\" (UniqueName: \"kubernetes.io/projected/ada10ddd-01bd-4521-b32d-872a8908d58d-kube-api-access-89c96\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.447892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.447855 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada10ddd-01bd-4521-b32d-872a8908d58d-openshift-service-ca\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.448058 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.447899 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.448058 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.447934 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89c96\" (UniqueName: \"kubernetes.io/projected/ada10ddd-01bd-4521-b32d-872a8908d58d-kube-api-access-89c96\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.448130 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:09:47.448067 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-serving-cert: secret "maas-api-e2e-mt-352df9d8-serving-cert" not found Jul 28 00:09:47.448130 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:09:47.448127 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls podName:ada10ddd-01bd-4521-b32d-872a8908d58d nodeName:}" failed. No retries permitted until 2026-07-28 00:09:47.948112452 +0000 UTC m=+1987.872321309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls") pod "maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" (UID: "ada10ddd-01bd-4521-b32d-872a8908d58d") : secret "maas-api-e2e-mt-352df9d8-serving-cert" not found Jul 28 00:09:47.448516 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.448483 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada10ddd-01bd-4521-b32d-872a8908d58d-openshift-service-ca\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.456064 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.456038 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89c96\" (UniqueName: \"kubernetes.io/projected/ada10ddd-01bd-4521-b32d-872a8908d58d-kube-api-access-89c96\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.722291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.722197 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:47.851324 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.851292 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:47.866626 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.866593 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg"] Jul 28 00:09:47.870172 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.870150 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:47.873703 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.873683 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-352df9d8\"" Jul 28 00:09:47.891645 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.891621 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg"] Jul 28 00:09:47.939507 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.939472 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g"] Jul 28 00:09:47.942772 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.942756 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:47.944979 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.944959 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-352df9d8-dockercfg-mbkjj\"" Jul 28 00:09:47.952057 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.952027 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a8b27f-603d-4784-b707-aa30dbfb22d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:47.952333 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.952072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55v4n\" (UniqueName: \"kubernetes.io/projected/b7a8b27f-603d-4784-b707-aa30dbfb22d0-kube-api-access-55v4n\") pod \"payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:47.952333 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.952135 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:47.954147 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.954112 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g"] Jul 28 00:09:47.954902 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:47.954879 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls\") pod \"maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:48.052946 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.052908 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70363da1-9bfe-4821-8a1d-4cca337c6358-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.053334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.052965 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7k9m\" (UniqueName: \"kubernetes.io/projected/70363da1-9bfe-4821-8a1d-4cca337c6358-kube-api-access-j7k9m\") pod \"payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.053334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.053041 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a8b27f-603d-4784-b707-aa30dbfb22d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:48.053334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.053085 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55v4n\" (UniqueName: \"kubernetes.io/projected/b7a8b27f-603d-4784-b707-aa30dbfb22d0-kube-api-access-55v4n\") pod \"payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:48.053787 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.053769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a8b27f-603d-4784-b707-aa30dbfb22d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:48.061181 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.061158 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55v4n\" (UniqueName: \"kubernetes.io/projected/b7a8b27f-603d-4784-b707-aa30dbfb22d0-kube-api-access-55v4n\") pod \"payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:48.154261 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.154222 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:48.154475 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.154449 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70363da1-9bfe-4821-8a1d-4cca337c6358-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.154588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.154542 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7k9m\" (UniqueName: \"kubernetes.io/projected/70363da1-9bfe-4821-8a1d-4cca337c6358-kube-api-access-j7k9m\") pod \"payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.155172 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.155152 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70363da1-9bfe-4821-8a1d-4cca337c6358-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.163537 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.163510 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7k9m\" (UniqueName: \"kubernetes.io/projected/70363da1-9bfe-4821-8a1d-4cca337c6358-kube-api-access-j7k9m\") pod \"payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.180250 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.180223 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:48.255574 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.253599 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.350278 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.350248 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg"] Jul 28 00:09:48.352573 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:09:48.352487 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a8b27f_603d_4784_b707_aa30dbfb22d0.slice/crio-85b7764a1f6e19a8367348999dfdbaa1133e47e4e7889a3629fa62d2a4ed9221 WatchSource:0}: Error finding container 85b7764a1f6e19a8367348999dfdbaa1133e47e4e7889a3629fa62d2a4ed9221: Status 404 returned error can't find the container with id 85b7764a1f6e19a8367348999dfdbaa1133e47e4e7889a3629fa62d2a4ed9221 Jul 28 00:09:48.405829 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.405806 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g"] Jul 28 00:09:48.409758 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:09:48.409732 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70363da1_9bfe_4821_8a1d_4cca337c6358.slice/crio-81723624d038b9ede8bf900f9512fbf1166013c3e5665208cb167e1d50a9c3c6 WatchSource:0}: Error finding container 81723624d038b9ede8bf900f9512fbf1166013c3e5665208cb167e1d50a9c3c6: Status 404 returned error can't find the container with id 81723624d038b9ede8bf900f9512fbf1166013c3e5665208cb167e1d50a9c3c6 Jul 28 00:09:48.541816 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:09:48.541781 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada10ddd_01bd_4521_b32d_872a8908d58d.slice/crio-24233c77d7872fd092d18d01b92c4c88c363ddcb484a37498b1e3f595c2c5390 WatchSource:0}: Error finding container 24233c77d7872fd092d18d01b92c4c88c363ddcb484a37498b1e3f595c2c5390: Status 404 returned error can't find the container with id 24233c77d7872fd092d18d01b92c4c88c363ddcb484a37498b1e3f595c2c5390 Jul 28 00:09:48.543338 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.543314 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:09:48.550541 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.550514 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5"] Jul 28 00:09:48.570875 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.570805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" event={"ID":"b7a8b27f-603d-4784-b707-aa30dbfb22d0","Type":"ContainerStarted","Data":"6d811b938145bf7056e0b09f167ff66ca72a4673333ed216014949eeb91ffb37"} Jul 28 00:09:48.570875 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.570855 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" event={"ID":"b7a8b27f-603d-4784-b707-aa30dbfb22d0","Type":"ContainerStarted","Data":"85b7764a1f6e19a8367348999dfdbaa1133e47e4e7889a3629fa62d2a4ed9221"} Jul 28 00:09:48.571014 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.570909 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:48.572350 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.572330 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" event={"ID":"70363da1-9bfe-4821-8a1d-4cca337c6358","Type":"ContainerStarted","Data":"c12410cb87124adc0d35e5eb3cd5d736cb5aa4e47dd00ad4db7bcea1b00fc5e0"} Jul 28 00:09:48.572449 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.572368 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" event={"ID":"70363da1-9bfe-4821-8a1d-4cca337c6358","Type":"ContainerStarted","Data":"81723624d038b9ede8bf900f9512fbf1166013c3e5665208cb167e1d50a9c3c6"} Jul 28 00:09:48.572449 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.572396 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:48.573598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.573576 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" event={"ID":"ada10ddd-01bd-4521-b32d-872a8908d58d","Type":"ContainerStarted","Data":"24233c77d7872fd092d18d01b92c4c88c363ddcb484a37498b1e3f595c2c5390"} Jul 28 00:09:48.607422 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.607371 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" podStartSLOduration=1.607353849 podStartE2EDuration="1.607353849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:09:48.605919278 +0000 UTC m=+1988.530128158" watchObservedRunningTime="2026-07-28 00:09:48.607353849 +0000 UTC m=+1988.531562730" Jul 28 00:09:48.638676 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:48.638626 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" podStartSLOduration=1.638609469 podStartE2EDuration="1.638609469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:09:48.637800269 +0000 UTC m=+1988.562009149" watchObservedRunningTime="2026-07-28 00:09:48.638609469 +0000 UTC m=+1988.562818346" Jul 28 00:09:49.581153 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:49.581107 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" event={"ID":"ada10ddd-01bd-4521-b32d-872a8908d58d","Type":"ContainerStarted","Data":"729f341b5b8c3feadd420bdb964d61b7e4a1bfd355859bb70fef71c14ca120e6"} Jul 28 00:09:49.581679 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:49.581340 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:49.597643 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:49.597600 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" podStartSLOduration=2.196863819 podStartE2EDuration="2.597587452s" totalImagesPullingTime="400.723633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:09:49.595631479 +0000 UTC m=+1989.519840357" watchObservedRunningTime="2026-07-28 00:09:49.597587452 +0000 UTC m=+1989.521796330" Jul 28 00:09:51.519310 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:51.519266 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:55.591487 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:55.591456 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:09:58.734527 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.734479 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4"] Jul 28 00:09:58.738846 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.738823 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.740916 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.740886 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-352df9d8\"/\"openshift-service-ca.crt\"" Jul 28 00:09:58.741699 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.741669 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-352df9d8\"/\"default-dockercfg-29w9m\"" Jul 28 00:09:58.741807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.741678 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-352df9d8\"/\"kube-root-ca.crt\"" Jul 28 00:09:58.741807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.741726 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-352df9d8\"/\"e2e-lifecycle-model-352df9d8-kserve-self-signed-certs\"" Jul 28 00:09:58.750051 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.749999 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4"] Jul 28 00:09:58.772764 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.772733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:58.845510 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.845468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668cba05-8d6e-4eac-be18-1c67e4225441-tls-certs\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.845653 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.845533 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-model-cache\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.845653 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.845605 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpv8\" (UniqueName: \"kubernetes.io/projected/668cba05-8d6e-4eac-be18-1c67e4225441-kube-api-access-bcpv8\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.845653 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.845631 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-dshm\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.845653 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.845647 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-tmp-dir\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.845829 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.845706 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-home\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.946774 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.946739 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-model-cache\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.946954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.946813 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpv8\" (UniqueName: \"kubernetes.io/projected/668cba05-8d6e-4eac-be18-1c67e4225441-kube-api-access-bcpv8\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.946954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.946852 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-dshm\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.946954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.946869 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-tmp-dir\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.947084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.947011 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-home\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.947084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.947067 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668cba05-8d6e-4eac-be18-1c67e4225441-tls-certs\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.947191 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.947150 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-model-cache\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.947253 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.947188 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-tmp-dir\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.947379 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.947359 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-home\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.949049 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.949030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-dshm\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.949443 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.949420 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668cba05-8d6e-4eac-be18-1c67e4225441-tls-certs\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:58.954783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:58.954759 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpv8\" (UniqueName: \"kubernetes.io/projected/668cba05-8d6e-4eac-be18-1c67e4225441-kube-api-access-bcpv8\") pod \"e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:59.052001 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.051966 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:09:59.186106 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.186055 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4"] Jul 28 00:09:59.191031 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:09:59.190997 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod668cba05_8d6e_4eac_be18_1c67e4225441.slice/crio-62f81d4a25deba9ce6e8a35ca1ea43f611d4fadb44bf0a48f8245ccc1de7ffa7 WatchSource:0}: Error finding container 62f81d4a25deba9ce6e8a35ca1ea43f611d4fadb44bf0a48f8245ccc1de7ffa7: Status 404 returned error can't find the container with id 62f81d4a25deba9ce6e8a35ca1ea43f611d4fadb44bf0a48f8245ccc1de7ffa7 Jul 28 00:09:59.583574 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.583535 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:09:59.583755 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.583637 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:09:59.637390 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.637346 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" event={"ID":"668cba05-8d6e-4eac-be18-1c67e4225441","Type":"ContainerStarted","Data":"e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32"} Jul 28 00:09:59.637571 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.637411 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" event={"ID":"668cba05-8d6e-4eac-be18-1c67e4225441","Type":"ContainerStarted","Data":"62f81d4a25deba9ce6e8a35ca1ea43f611d4fadb44bf0a48f8245ccc1de7ffa7"} Jul 28 00:09:59.654866 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:09:59.654811 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" podStartSLOduration=1.654792429 podStartE2EDuration="1.654792429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:09:59.651863412 +0000 UTC m=+1999.576072283" watchObservedRunningTime="2026-07-28 00:09:59.654792429 +0000 UTC m=+1999.579001307" Jul 28 00:10:09.052994 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:09.052953 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:10:09.053644 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:09.053016 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:10:09.065997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:09.065970 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:10:09.696889 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:09.696857 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:10:26.156549 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.156512 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6"] Jul 28 00:10:26.159390 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.159365 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:26.161590 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.161568 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:10:26.167286 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.167089 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6"] Jul 28 00:10:26.287092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.287053 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58b588-fc7f-4720-9ea4-9b04ea6afffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-352df9d8-swdl6\" (UID: \"6d58b588-fc7f-4720-9ea4-9b04ea6afffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:26.388397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.388355 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58b588-fc7f-4720-9ea4-9b04ea6afffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-352df9d8-swdl6\" (UID: \"6d58b588-fc7f-4720-9ea4-9b04ea6afffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:26.389029 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.389009 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58b588-fc7f-4720-9ea4-9b04ea6afffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-352df9d8-swdl6\" (UID: \"6d58b588-fc7f-4720-9ea4-9b04ea6afffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:26.471641 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.471559 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:26.610960 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.610931 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6"] Jul 28 00:10:26.612658 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:10:26.612624 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d58b588_fc7f_4720_9ea4_9b04ea6afffd.slice/crio-73ec341846afebe576479351a42889b291f2e75284ea5471f3cc7fd573a3f913 WatchSource:0}: Error finding container 73ec341846afebe576479351a42889b291f2e75284ea5471f3cc7fd573a3f913: Status 404 returned error can't find the container with id 73ec341846afebe576479351a42889b291f2e75284ea5471f3cc7fd573a3f913 Jul 28 00:10:26.780594 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:26.780558 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" event={"ID":"6d58b588-fc7f-4720-9ea4-9b04ea6afffd","Type":"ContainerStarted","Data":"73ec341846afebe576479351a42889b291f2e75284ea5471f3cc7fd573a3f913"} Jul 28 00:10:27.788383 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:27.788338 2555 generic.go:356] "Generic (PLEG): container finished" podID="6d58b588-fc7f-4720-9ea4-9b04ea6afffd" containerID="31817204b883e839080f65125c606e9f7b4c17f773d58e086df6c7633b60ddee" exitCode=0 Jul 28 00:10:27.788859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:27.788428 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" event={"ID":"6d58b588-fc7f-4720-9ea4-9b04ea6afffd","Type":"ContainerDied","Data":"31817204b883e839080f65125c606e9f7b4c17f773d58e086df6c7633b60ddee"} Jul 28 00:10:28.928164 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:28.928139 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:29.016229 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:29.016194 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58b588-fc7f-4720-9ea4-9b04ea6afffd-maas-api-service-ca\") pod \"6d58b588-fc7f-4720-9ea4-9b04ea6afffd\" (UID: \"6d58b588-fc7f-4720-9ea4-9b04ea6afffd\") " Jul 28 00:10:29.016618 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:29.016594 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d58b588-fc7f-4720-9ea4-9b04ea6afffd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d58b588-fc7f-4720-9ea4-9b04ea6afffd" (UID: "6d58b588-fc7f-4720-9ea4-9b04ea6afffd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:10:29.117157 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:29.117066 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58b588-fc7f-4720-9ea4-9b04ea6afffd-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:29.799945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:29.799915 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" Jul 28 00:10:29.799945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:29.799924 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6" event={"ID":"6d58b588-fc7f-4720-9ea4-9b04ea6afffd","Type":"ContainerDied","Data":"73ec341846afebe576479351a42889b291f2e75284ea5471f3cc7fd573a3f913"} Jul 28 00:10:29.800174 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:29.799969 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ec341846afebe576479351a42889b291f2e75284ea5471f3cc7fd573a3f913" Jul 28 00:10:41.614041 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.614007 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5"] Jul 28 00:10:41.614624 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.614235 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" podUID="ada10ddd-01bd-4521-b32d-872a8908d58d" containerName="maas-api" containerID="cri-o://729f341b5b8c3feadd420bdb964d61b7e4a1bfd355859bb70fef71c14ca120e6" gracePeriod=30 Jul 28 00:10:41.655170 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.655118 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:41.692738 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.692708 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg"] Jul 28 00:10:41.692956 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.692928 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" podUID="b7a8b27f-603d-4784-b707-aa30dbfb22d0" containerName="payload-processing" containerID="cri-o://6d811b938145bf7056e0b09f167ff66ca72a4673333ed216014949eeb91ffb37" gracePeriod=30 Jul 28 00:10:41.713376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.712193 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g"] Jul 28 00:10:41.713376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.712458 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" podUID="70363da1-9bfe-4821-8a1d-4cca337c6358" containerName="payload-pre-processing" containerID="cri-o://c12410cb87124adc0d35e5eb3cd5d736cb5aa4e47dd00ad4db7bcea1b00fc5e0" gracePeriod=30 Jul 28 00:10:41.755806 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.755760 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:41.854823 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.854739 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:41.870539 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.870411 2555 generic.go:356] "Generic (PLEG): container finished" podID="b7a8b27f-603d-4784-b707-aa30dbfb22d0" containerID="6d811b938145bf7056e0b09f167ff66ca72a4673333ed216014949eeb91ffb37" exitCode=0 Jul 28 00:10:41.870728 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.870540 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" event={"ID":"b7a8b27f-603d-4784-b707-aa30dbfb22d0","Type":"ContainerDied","Data":"6d811b938145bf7056e0b09f167ff66ca72a4673333ed216014949eeb91ffb37"} Jul 28 00:10:41.873026 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.872993 2555 generic.go:356] "Generic (PLEG): container finished" podID="70363da1-9bfe-4821-8a1d-4cca337c6358" containerID="c12410cb87124adc0d35e5eb3cd5d736cb5aa4e47dd00ad4db7bcea1b00fc5e0" exitCode=0 Jul 28 00:10:41.873161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.873079 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" event={"ID":"70363da1-9bfe-4821-8a1d-4cca337c6358","Type":"ContainerDied","Data":"c12410cb87124adc0d35e5eb3cd5d736cb5aa4e47dd00ad4db7bcea1b00fc5e0"} Jul 28 00:10:41.875539 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.875404 2555 generic.go:356] "Generic (PLEG): container finished" podID="ada10ddd-01bd-4521-b32d-872a8908d58d" containerID="729f341b5b8c3feadd420bdb964d61b7e4a1bfd355859bb70fef71c14ca120e6" exitCode=0 Jul 28 00:10:41.875539 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.875469 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" event={"ID":"ada10ddd-01bd-4521-b32d-872a8908d58d","Type":"ContainerDied","Data":"729f341b5b8c3feadd420bdb964d61b7e4a1bfd355859bb70fef71c14ca120e6"} Jul 28 00:10:41.956663 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.956625 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:10:41.994159 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:41.994130 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:10:42.004736 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.004707 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:10:42.138395 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138305 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70363da1-9bfe-4821-8a1d-4cca337c6358-plugins-config-volume\") pod \"70363da1-9bfe-4821-8a1d-4cca337c6358\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " Jul 28 00:10:42.138395 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138374 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada10ddd-01bd-4521-b32d-872a8908d58d-openshift-service-ca\") pod \"ada10ddd-01bd-4521-b32d-872a8908d58d\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " Jul 28 00:10:42.138684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138424 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55v4n\" (UniqueName: \"kubernetes.io/projected/b7a8b27f-603d-4784-b707-aa30dbfb22d0-kube-api-access-55v4n\") pod \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " Jul 28 00:10:42.138684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138459 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a8b27f-603d-4784-b707-aa30dbfb22d0-plugins-config-volume\") pod \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\" (UID: \"b7a8b27f-603d-4784-b707-aa30dbfb22d0\") " Jul 28 00:10:42.138684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138579 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7k9m\" (UniqueName: \"kubernetes.io/projected/70363da1-9bfe-4821-8a1d-4cca337c6358-kube-api-access-j7k9m\") pod \"70363da1-9bfe-4821-8a1d-4cca337c6358\" (UID: \"70363da1-9bfe-4821-8a1d-4cca337c6358\") " Jul 28 00:10:42.138684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138637 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89c96\" (UniqueName: \"kubernetes.io/projected/ada10ddd-01bd-4521-b32d-872a8908d58d-kube-api-access-89c96\") pod \"ada10ddd-01bd-4521-b32d-872a8908d58d\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " Jul 28 00:10:42.138684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138678 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls\") pod \"ada10ddd-01bd-4521-b32d-872a8908d58d\" (UID: \"ada10ddd-01bd-4521-b32d-872a8908d58d\") " Jul 28 00:10:42.138960 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138801 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70363da1-9bfe-4821-8a1d-4cca337c6358-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70363da1-9bfe-4821-8a1d-4cca337c6358" (UID: "70363da1-9bfe-4821-8a1d-4cca337c6358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:10:42.138960 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138809 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a8b27f-603d-4784-b707-aa30dbfb22d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7a8b27f-603d-4784-b707-aa30dbfb22d0" (UID: "b7a8b27f-603d-4784-b707-aa30dbfb22d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:10:42.138960 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.138815 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada10ddd-01bd-4521-b32d-872a8908d58d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ada10ddd-01bd-4521-b32d-872a8908d58d" (UID: "ada10ddd-01bd-4521-b32d-872a8908d58d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:10:42.139114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.139004 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70363da1-9bfe-4821-8a1d-4cca337c6358-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.139114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.139023 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada10ddd-01bd-4521-b32d-872a8908d58d-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.139114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.139035 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a8b27f-603d-4784-b707-aa30dbfb22d0-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.140723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.140698 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ada10ddd-01bd-4521-b32d-872a8908d58d" (UID: "ada10ddd-01bd-4521-b32d-872a8908d58d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:10:42.140850 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.140820 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a8b27f-603d-4784-b707-aa30dbfb22d0-kube-api-access-55v4n" (OuterVolumeSpecName: "kube-api-access-55v4n") pod "b7a8b27f-603d-4784-b707-aa30dbfb22d0" (UID: "b7a8b27f-603d-4784-b707-aa30dbfb22d0"). InnerVolumeSpecName "kube-api-access-55v4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:10:42.140910 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.140854 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70363da1-9bfe-4821-8a1d-4cca337c6358-kube-api-access-j7k9m" (OuterVolumeSpecName: "kube-api-access-j7k9m") pod "70363da1-9bfe-4821-8a1d-4cca337c6358" (UID: "70363da1-9bfe-4821-8a1d-4cca337c6358"). InnerVolumeSpecName "kube-api-access-j7k9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:10:42.140910 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.140881 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada10ddd-01bd-4521-b32d-872a8908d58d-kube-api-access-89c96" (OuterVolumeSpecName: "kube-api-access-89c96") pod "ada10ddd-01bd-4521-b32d-872a8908d58d" (UID: "ada10ddd-01bd-4521-b32d-872a8908d58d"). InnerVolumeSpecName "kube-api-access-89c96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:10:42.239839 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.239794 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55v4n\" (UniqueName: \"kubernetes.io/projected/b7a8b27f-603d-4784-b707-aa30dbfb22d0-kube-api-access-55v4n\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.239839 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.239835 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7k9m\" (UniqueName: \"kubernetes.io/projected/70363da1-9bfe-4821-8a1d-4cca337c6358-kube-api-access-j7k9m\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.239839 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.239845 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89c96\" (UniqueName: \"kubernetes.io/projected/ada10ddd-01bd-4521-b32d-872a8908d58d-kube-api-access-89c96\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.239839 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.239854 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada10ddd-01bd-4521-b32d-872a8908d58d-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:10:42.882990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.882915 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" Jul 28 00:10:42.883438 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.882907 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg" event={"ID":"b7a8b27f-603d-4784-b707-aa30dbfb22d0","Type":"ContainerDied","Data":"85b7764a1f6e19a8367348999dfdbaa1133e47e4e7889a3629fa62d2a4ed9221"} Jul 28 00:10:42.883438 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.883078 2555 scope.go:117] "RemoveContainer" containerID="6d811b938145bf7056e0b09f167ff66ca72a4673333ed216014949eeb91ffb37" Jul 28 00:10:42.884834 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.884476 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" event={"ID":"70363da1-9bfe-4821-8a1d-4cca337c6358","Type":"ContainerDied","Data":"81723624d038b9ede8bf900f9512fbf1166013c3e5665208cb167e1d50a9c3c6"} Jul 28 00:10:42.884834 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.884505 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g" Jul 28 00:10:42.886391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.886367 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" Jul 28 00:10:42.886484 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.886403 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5" event={"ID":"ada10ddd-01bd-4521-b32d-872a8908d58d","Type":"ContainerDied","Data":"24233c77d7872fd092d18d01b92c4c88c363ddcb484a37498b1e3f595c2c5390"} Jul 28 00:10:42.893152 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.893135 2555 scope.go:117] "RemoveContainer" containerID="c12410cb87124adc0d35e5eb3cd5d736cb5aa4e47dd00ad4db7bcea1b00fc5e0" Jul 28 00:10:42.900832 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.900809 2555 scope.go:117] "RemoveContainer" containerID="729f341b5b8c3feadd420bdb964d61b7e4a1bfd355859bb70fef71c14ca120e6" Jul 28 00:10:42.916543 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.916516 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg"] Jul 28 00:10:42.921875 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.921846 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-352df9d8-7bc565bc86-lrtpg"] Jul 28 00:10:42.950038 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.950013 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g"] Jul 28 00:10:42.964858 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.964827 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-352df9d8-6674ffdffb-nml6g"] Jul 28 00:10:42.988323 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.988292 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5"] Jul 28 00:10:42.995827 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:42.995800 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-352df9d8-f77f77f4b-nhfv5"] Jul 28 00:10:44.565518 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:44.565472 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70363da1-9bfe-4821-8a1d-4cca337c6358" path="/var/lib/kubelet/pods/70363da1-9bfe-4821-8a1d-4cca337c6358/volumes" Jul 28 00:10:44.565919 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:44.565877 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada10ddd-01bd-4521-b32d-872a8908d58d" path="/var/lib/kubelet/pods/ada10ddd-01bd-4521-b32d-872a8908d58d/volumes" Jul 28 00:10:44.566189 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:44.566172 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a8b27f-603d-4784-b707-aa30dbfb22d0" path="/var/lib/kubelet/pods/b7a8b27f-603d-4784-b707-aa30dbfb22d0/volumes" Jul 28 00:10:46.236088 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:46.236049 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:46.241225 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:46.241199 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6"] Jul 28 00:10:46.243856 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:46.243835 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-352df9d8-swdl6"] Jul 28 00:10:46.565271 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:46.565240 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d58b588-fc7f-4720-9ea4-9b04ea6afffd" path="/var/lib/kubelet/pods/6d58b588-fc7f-4720-9ea4-9b04ea6afffd/volumes" Jul 28 00:10:49.668902 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:49.668863 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:51.832449 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:51.832412 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:54.650865 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:54.650824 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4"] Jul 28 00:10:54.746583 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:10:54.746544 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:09.670479 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:09.670380 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" podUID="668cba05-8d6e-4eac-be18-1c67e4225441" containerName="main" containerID="cri-o://e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32" gracePeriod=45 Jul 28 00:11:09.686435 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:09.686400 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" podUID="668cba05-8d6e-4eac-be18-1c67e4225441" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": dial tcp 10.133.0.73:8000: connect: connection refused" Jul 28 00:11:09.928279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:09.928206 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:11:10.028889 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.028854 2555 generic.go:356] "Generic (PLEG): container finished" podID="668cba05-8d6e-4eac-be18-1c67e4225441" containerID="e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32" exitCode=0 Jul 28 00:11:10.029084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.028900 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" Jul 28 00:11:10.029084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.028940 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" event={"ID":"668cba05-8d6e-4eac-be18-1c67e4225441","Type":"ContainerDied","Data":"e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32"} Jul 28 00:11:10.029084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.028985 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4" event={"ID":"668cba05-8d6e-4eac-be18-1c67e4225441","Type":"ContainerDied","Data":"62f81d4a25deba9ce6e8a35ca1ea43f611d4fadb44bf0a48f8245ccc1de7ffa7"} Jul 28 00:11:10.029084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.029005 2555 scope.go:117] "RemoveContainer" containerID="e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32" Jul 28 00:11:10.037679 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.037657 2555 scope.go:117] "RemoveContainer" containerID="e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32" Jul 28 00:11:10.037954 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:11:10.037935 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32\": container with ID starting with e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32 not found: ID does not exist" containerID="e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32" Jul 28 00:11:10.038019 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.037969 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32"} err="failed to get container status \"e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32\": rpc error: code = NotFound desc = could not find container \"e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32\": container with ID starting with e61a2883d5225211f40fa22b6a16403dbc3c1224433e2eb01622699f4b7a2c32 not found: ID does not exist" Jul 28 00:11:10.074279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074236 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-model-cache\") pod \"668cba05-8d6e-4eac-be18-1c67e4225441\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " Jul 28 00:11:10.074279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074278 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668cba05-8d6e-4eac-be18-1c67e4225441-tls-certs\") pod \"668cba05-8d6e-4eac-be18-1c67e4225441\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " Jul 28 00:11:10.074571 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074297 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcpv8\" (UniqueName: \"kubernetes.io/projected/668cba05-8d6e-4eac-be18-1c67e4225441-kube-api-access-bcpv8\") pod \"668cba05-8d6e-4eac-be18-1c67e4225441\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " Jul 28 00:11:10.074571 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074370 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-home\") pod \"668cba05-8d6e-4eac-be18-1c67e4225441\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " Jul 28 00:11:10.074571 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074401 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-dshm\") pod \"668cba05-8d6e-4eac-be18-1c67e4225441\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " Jul 28 00:11:10.074571 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074454 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-tmp-dir\") pod \"668cba05-8d6e-4eac-be18-1c67e4225441\" (UID: \"668cba05-8d6e-4eac-be18-1c67e4225441\") " Jul 28 00:11:10.074797 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074569 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-model-cache" (OuterVolumeSpecName: "model-cache") pod "668cba05-8d6e-4eac-be18-1c67e4225441" (UID: "668cba05-8d6e-4eac-be18-1c67e4225441"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:11:10.074797 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074678 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-home" (OuterVolumeSpecName: "home") pod "668cba05-8d6e-4eac-be18-1c67e4225441" (UID: "668cba05-8d6e-4eac-be18-1c67e4225441"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:11:10.074797 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074754 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "668cba05-8d6e-4eac-be18-1c67e4225441" (UID: "668cba05-8d6e-4eac-be18-1c67e4225441"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:11:10.074945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074815 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:11:10.074945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074832 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:11:10.074945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.074845 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:11:10.076663 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.076637 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668cba05-8d6e-4eac-be18-1c67e4225441-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "668cba05-8d6e-4eac-be18-1c67e4225441" (UID: "668cba05-8d6e-4eac-be18-1c67e4225441"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:11:10.076912 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.076883 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-dshm" (OuterVolumeSpecName: "dshm") pod "668cba05-8d6e-4eac-be18-1c67e4225441" (UID: "668cba05-8d6e-4eac-be18-1c67e4225441"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:11:10.077006 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.076913 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668cba05-8d6e-4eac-be18-1c67e4225441-kube-api-access-bcpv8" (OuterVolumeSpecName: "kube-api-access-bcpv8") pod "668cba05-8d6e-4eac-be18-1c67e4225441" (UID: "668cba05-8d6e-4eac-be18-1c67e4225441"). InnerVolumeSpecName "kube-api-access-bcpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:11:10.175483 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.175441 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668cba05-8d6e-4eac-be18-1c67e4225441-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:11:10.175483 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.175477 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668cba05-8d6e-4eac-be18-1c67e4225441-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:11:10.175483 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.175489 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpv8\" (UniqueName: \"kubernetes.io/projected/668cba05-8d6e-4eac-be18-1c67e4225441-kube-api-access-bcpv8\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:11:10.348392 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.348350 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4"] Jul 28 00:11:10.352244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.352212 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-352df9d8/e2e-lifecycle-model-352df9d8-kserve-d8bcc8895-sz8f4"] Jul 28 00:11:10.564992 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:10.564954 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668cba05-8d6e-4eac-be18-1c67e4225441" path="/var/lib/kubelet/pods/668cba05-8d6e-4eac-be18-1c67e4225441/volumes" Jul 28 00:11:30.948237 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:30.948201 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:31.710004 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:31.709965 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:37.325667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:37.325631 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:40.914015 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:40.913980 2555 scope.go:117] "RemoveContainer" containerID="7f04446f3fcd3e9e49f179dda9b58ab7b6f6a68d8b6ee322a2aeb78c81eee996" Jul 28 00:11:40.921897 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:40.921876 2555 scope.go:117] "RemoveContainer" containerID="6d44066c5c0863dc5ef35552186300a6903aa8ad713d40282145cec27f1c1597" Jul 28 00:11:42.964688 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:42.964649 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:43.064987 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:43.064944 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:43.230195 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:43.230109 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:43.493747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:43.493706 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:44.652046 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:44.652014 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:44.753312 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:44.753270 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:44.865024 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:44.864982 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:45.633713 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:45.633667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:45.725783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:45.725751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:45.825957 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:45.825922 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:47.326300 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:47.326256 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:52.987746 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:52.987710 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:53.088733 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:53.088701 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:53.690567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:11:53.690530 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:00.341508 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:00.341456 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:00.450213 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:00.450179 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:00.541850 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:00.541814 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:11.079277 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:11.079236 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:11.427471 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:11.427373 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:32.439432 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:32.439391 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:38.844719 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:38.844680 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:38.947716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:38.947654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:39.217537 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:39.217421 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:39.558026 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:39.557989 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:39.959588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:39.959474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:40.964281 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:40.964246 2555 scope.go:117] "RemoveContainer" containerID="f6e35cb508fd9000f08bf14f224a6ea444b687d035afafbc3fbd1927a6ea723c" Jul 28 00:12:43.026848 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.026802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:43.113764 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.113728 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j"] Jul 28 00:12:43.114283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114263 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a8b27f-603d-4784-b707-aa30dbfb22d0" containerName="payload-processing" Jul 28 00:12:43.114283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114283 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a8b27f-603d-4784-b707-aa30dbfb22d0" containerName="payload-processing" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114303 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d58b588-fc7f-4720-9ea4-9b04ea6afffd" containerName="revoke-keys" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114313 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d58b588-fc7f-4720-9ea4-9b04ea6afffd" containerName="revoke-keys" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114334 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada10ddd-01bd-4521-b32d-872a8908d58d" containerName="maas-api" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114344 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada10ddd-01bd-4521-b32d-872a8908d58d" containerName="maas-api" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114354 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70363da1-9bfe-4821-8a1d-4cca337c6358" containerName="payload-pre-processing" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114362 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70363da1-9bfe-4821-8a1d-4cca337c6358" containerName="payload-pre-processing" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114374 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668cba05-8d6e-4eac-be18-1c67e4225441" containerName="main" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114382 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="668cba05-8d6e-4eac-be18-1c67e4225441" containerName="main" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114456 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="70363da1-9bfe-4821-8a1d-4cca337c6358" containerName="payload-pre-processing" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114468 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a8b27f-603d-4784-b707-aa30dbfb22d0" containerName="payload-processing" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114480 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada10ddd-01bd-4521-b32d-872a8908d58d" containerName="maas-api" Jul 28 00:12:43.114515 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114488 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d58b588-fc7f-4720-9ea4-9b04ea6afffd" containerName="revoke-keys" Jul 28 00:12:43.115118 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114533 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="668cba05-8d6e-4eac-be18-1c67e4225441" containerName="main" Jul 28 00:12:43.115118 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.114893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:43.118042 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.118022 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.120458 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.120438 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f7352d-serving-cert\"" Jul 28 00:12:43.126950 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.126917 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j"] Jul 28 00:12:43.181880 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.181840 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:43.184872 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.184844 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x65g\" (UniqueName: \"kubernetes.io/projected/587044f1-2d96-48d2-8517-8590d282b7c6-kube-api-access-6x65g\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.184997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.184903 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.184997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.184920 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587044f1-2d96-48d2-8517-8590d282b7c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.286440 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.286349 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x65g\" (UniqueName: \"kubernetes.io/projected/587044f1-2d96-48d2-8517-8590d282b7c6-kube-api-access-6x65g\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.286629 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.286532 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.286629 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.286565 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587044f1-2d96-48d2-8517-8590d282b7c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.286752 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:12:43.286655 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-serving-cert: secret "maas-api-e2e-inf-a-f7352d-serving-cert" not found Jul 28 00:12:43.286752 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:12:43.286725 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls podName:587044f1-2d96-48d2-8517-8590d282b7c6 nodeName:}" failed. No retries permitted until 2026-07-28 00:12:43.786705351 +0000 UTC m=+2163.710914211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls") pod "maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" (UID: "587044f1-2d96-48d2-8517-8590d282b7c6") : secret "maas-api-e2e-inf-a-f7352d-serving-cert" not found Jul 28 00:12:43.287161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.287143 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587044f1-2d96-48d2-8517-8590d282b7c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.295424 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.295405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x65g\" (UniqueName: \"kubernetes.io/projected/587044f1-2d96-48d2-8517-8590d282b7c6-kube-api-access-6x65g\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.364022 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.363983 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:43.792227 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.792193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.794598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.794568 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-f7352d-7985555786-d7v4j\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:43.802337 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.802306 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:43.891927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.891889 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8"] Jul 28 00:12:43.895454 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.895432 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:43.896400 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.896361 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:43.897771 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.897753 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f7352d\"" Jul 28 00:12:43.907085 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.907059 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8"] Jul 28 00:12:43.934218 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.934189 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf"] Jul 28 00:12:43.937748 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.937732 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:43.942307 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.942286 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f7352d-dockercfg-l4tlr\"" Jul 28 00:12:43.951444 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.951424 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf"] Jul 28 00:12:43.993860 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.993828 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93a80ed-5422-42c3-a577-95824ddfd574-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:43.993999 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.993864 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw29t\" (UniqueName: \"kubernetes.io/projected/f93a80ed-5422-42c3-a577-95824ddfd574-kube-api-access-kw29t\") pod \"payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:43.993999 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.993891 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcns\" (UniqueName: \"kubernetes.io/projected/ace5567c-6162-4bea-af52-15f883238f42-kube-api-access-dtcns\") pod \"payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:43.994090 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:43.993998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace5567c-6162-4bea-af52-15f883238f42-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.030963 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.030934 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:44.095148 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.095106 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace5567c-6162-4bea-af52-15f883238f42-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.095320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.095204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93a80ed-5422-42c3-a577-95824ddfd574-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:44.095320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.095258 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw29t\" (UniqueName: \"kubernetes.io/projected/f93a80ed-5422-42c3-a577-95824ddfd574-kube-api-access-kw29t\") pod \"payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:44.095320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.095298 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcns\" (UniqueName: \"kubernetes.io/projected/ace5567c-6162-4bea-af52-15f883238f42-kube-api-access-dtcns\") pod \"payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.096477 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.096441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace5567c-6162-4bea-af52-15f883238f42-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.097221 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.097192 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93a80ed-5422-42c3-a577-95824ddfd574-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:44.105485 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.105446 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcns\" (UniqueName: \"kubernetes.io/projected/ace5567c-6162-4bea-af52-15f883238f42-kube-api-access-dtcns\") pod \"payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.108767 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.108742 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw29t\" (UniqueName: \"kubernetes.io/projected/f93a80ed-5422-42c3-a577-95824ddfd574-kube-api-access-kw29t\") pod \"payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:44.171659 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.171634 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j"] Jul 28 00:12:44.173688 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:12:44.173660 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587044f1_2d96_48d2_8517_8590d282b7c6.slice/crio-9fb278ec8b05725d4a1cdc77af702c43746aca69b3d39c978e6fddd911d91ca8 WatchSource:0}: Error finding container 9fb278ec8b05725d4a1cdc77af702c43746aca69b3d39c978e6fddd911d91ca8: Status 404 returned error can't find the container with id 9fb278ec8b05725d4a1cdc77af702c43746aca69b3d39c978e6fddd911d91ca8 Jul 28 00:12:44.212217 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.212182 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:44.248762 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.248726 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.402677 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.402614 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf"] Jul 28 00:12:44.405139 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:12:44.405113 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace5567c_6162_4bea_af52_15f883238f42.slice/crio-834fee2afea5ef68de497468b146212b25449fba65194e686df3c02d95dcee09 WatchSource:0}: Error finding container 834fee2afea5ef68de497468b146212b25449fba65194e686df3c02d95dcee09: Status 404 returned error can't find the container with id 834fee2afea5ef68de497468b146212b25449fba65194e686df3c02d95dcee09 Jul 28 00:12:44.499092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.499057 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" event={"ID":"ace5567c-6162-4bea-af52-15f883238f42","Type":"ContainerStarted","Data":"1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb"} Jul 28 00:12:44.499270 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.499226 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:44.499339 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.499294 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" event={"ID":"ace5567c-6162-4bea-af52-15f883238f42","Type":"ContainerStarted","Data":"834fee2afea5ef68de497468b146212b25449fba65194e686df3c02d95dcee09"} Jul 28 00:12:44.500348 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.500313 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" event={"ID":"587044f1-2d96-48d2-8517-8590d282b7c6","Type":"ContainerStarted","Data":"9fb278ec8b05725d4a1cdc77af702c43746aca69b3d39c978e6fddd911d91ca8"} Jul 28 00:12:44.515165 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.515118 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" podStartSLOduration=1.515101327 podStartE2EDuration="1.515101327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:12:44.512724173 +0000 UTC m=+2164.436933051" watchObservedRunningTime="2026-07-28 00:12:44.515101327 +0000 UTC m=+2164.439310205" Jul 28 00:12:44.573366 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:44.573342 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8"] Jul 28 00:12:44.575731 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:12:44.575701 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93a80ed_5422_42c3_a577_95824ddfd574.slice/crio-8a9b55bb2bacfef43602624938a364940b0052d003659d53c7aa052e058efe9c WatchSource:0}: Error finding container 8a9b55bb2bacfef43602624938a364940b0052d003659d53c7aa052e058efe9c: Status 404 returned error can't find the container with id 8a9b55bb2bacfef43602624938a364940b0052d003659d53c7aa052e058efe9c Jul 28 00:12:45.508103 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:45.508070 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" event={"ID":"f93a80ed-5422-42c3-a577-95824ddfd574","Type":"ContainerStarted","Data":"58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee"} Jul 28 00:12:45.508576 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:45.508123 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" event={"ID":"f93a80ed-5422-42c3-a577-95824ddfd574","Type":"ContainerStarted","Data":"8a9b55bb2bacfef43602624938a364940b0052d003659d53c7aa052e058efe9c"} Jul 28 00:12:45.508576 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:45.508162 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:45.509588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:45.509559 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" event={"ID":"587044f1-2d96-48d2-8517-8590d282b7c6","Type":"ContainerStarted","Data":"58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0"} Jul 28 00:12:45.525112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:45.525062 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" podStartSLOduration=2.525048248 podStartE2EDuration="2.525048248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:12:45.522792901 +0000 UTC m=+2165.447001782" watchObservedRunningTime="2026-07-28 00:12:45.525048248 +0000 UTC m=+2165.449257126" Jul 28 00:12:45.539476 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:45.539430 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" podStartSLOduration=1.9042619250000001 podStartE2EDuration="2.539415179s" totalImagesPullingTime="635.153254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:12:45.538310651 +0000 UTC m=+2165.462519532" watchObservedRunningTime="2026-07-28 00:12:45.539415179 +0000 UTC m=+2165.463624056" Jul 28 00:12:46.515679 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:46.515647 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:47.264936 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:47.264900 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:47.599903 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:47.599805 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:47.978242 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:47.978153 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:48.299432 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:48.299392 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:49.112783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:49.112742 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.051547 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.051483 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.137784 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.137738 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.142667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.142641 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb"] Jul 28 00:12:51.147629 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.147609 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.149581 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.149556 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2ccf60-serving-cert\"" Jul 28 00:12:51.157557 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.157535 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb"] Jul 28 00:12:51.210790 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.210760 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.257788 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.257753 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dad814-b19e-481f-b555-14c6733958ca-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.257963 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.257860 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhrx\" (UniqueName: \"kubernetes.io/projected/d9dad814-b19e-481f-b555-14c6733958ca-kube-api-access-9dhrx\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.257963 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.257894 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9dad814-b19e-481f-b555-14c6733958ca-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.358407 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.358317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dad814-b19e-481f-b555-14c6733958ca-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.358407 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.358402 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhrx\" (UniqueName: \"kubernetes.io/projected/d9dad814-b19e-481f-b555-14c6733958ca-kube-api-access-9dhrx\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.358626 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.358429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9dad814-b19e-481f-b555-14c6733958ca-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.359114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.359093 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dad814-b19e-481f-b555-14c6733958ca-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.360908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.360889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9dad814-b19e-481f-b555-14c6733958ca-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.365522 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.365484 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhrx\" (UniqueName: \"kubernetes.io/projected/d9dad814-b19e-481f-b555-14c6733958ca-kube-api-access-9dhrx\") pod \"maas-api-e2e-inf-b-2ccf60-75df949c84-skspb\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.368654 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.368629 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.461787 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.461749 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:51.593830 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.593805 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb"] Jul 28 00:12:51.596923 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:12:51.596891 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dad814_b19e_481f_b555_14c6733958ca.slice/crio-4f4d9d6a5142b665fc836e5054cd86cf49b39370b0a7a511bceb95c554b7ab62 WatchSource:0}: Error finding container 4f4d9d6a5142b665fc836e5054cd86cf49b39370b0a7a511bceb95c554b7ab62: Status 404 returned error can't find the container with id 4f4d9d6a5142b665fc836e5054cd86cf49b39370b0a7a511bceb95c554b7ab62 Jul 28 00:12:51.723128 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.723029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.814182 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.814142 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:51.817995 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.817970 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r"] Jul 28 00:12:51.822631 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.822611 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:51.824779 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.824762 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2ccf60-dockercfg-7lzdp\"" Jul 28 00:12:51.824874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.824795 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2ccf60\"" Jul 28 00:12:51.832403 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.832372 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r"] Jul 28 00:12:51.850507 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.850460 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22"] Jul 28 00:12:51.855188 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.855165 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:51.863325 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.863300 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22"] Jul 28 00:12:51.962537 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.962480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmp9w\" (UniqueName: \"kubernetes.io/projected/58be2baf-7afb-44d2-a659-e94fb5b94d8a-kube-api-access-kmp9w\") pod \"payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:51.962716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.962594 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5639af3a-9df3-4991-954d-edb789bf43b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:51.962716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.962673 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58be2baf-7afb-44d2-a659-e94fb5b94d8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:51.962818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:51.962750 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgvf\" (UniqueName: \"kubernetes.io/projected/5639af3a-9df3-4991-954d-edb789bf43b9-kube-api-access-2tgvf\") pod \"payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:52.063331 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.063300 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmp9w\" (UniqueName: \"kubernetes.io/projected/58be2baf-7afb-44d2-a659-e94fb5b94d8a-kube-api-access-kmp9w\") pod \"payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:52.063756 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.063364 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5639af3a-9df3-4991-954d-edb789bf43b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:52.063756 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.063433 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58be2baf-7afb-44d2-a659-e94fb5b94d8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:52.063756 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.063553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgvf\" (UniqueName: \"kubernetes.io/projected/5639af3a-9df3-4991-954d-edb789bf43b9-kube-api-access-2tgvf\") pod \"payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:52.064101 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.064079 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5639af3a-9df3-4991-954d-edb789bf43b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:52.064257 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.064234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58be2baf-7afb-44d2-a659-e94fb5b94d8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:52.070747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.070718 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmp9w\" (UniqueName: \"kubernetes.io/projected/58be2baf-7afb-44d2-a659-e94fb5b94d8a-kube-api-access-kmp9w\") pod \"payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:52.070826 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.070761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgvf\" (UniqueName: \"kubernetes.io/projected/5639af3a-9df3-4991-954d-edb789bf43b9-kube-api-access-2tgvf\") pod \"payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:52.135087 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.135063 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:52.166128 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.166100 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:52.325906 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.325797 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22"] Jul 28 00:12:52.328150 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:12:52.328120 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58be2baf_7afb_44d2_a659_e94fb5b94d8a.slice/crio-665d2be6ee6e3ce6d0105e78e4d8b0f9ef9df9fb01c5c828103deb9c57bcda90 WatchSource:0}: Error finding container 665d2be6ee6e3ce6d0105e78e4d8b0f9ef9df9fb01c5c828103deb9c57bcda90: Status 404 returned error can't find the container with id 665d2be6ee6e3ce6d0105e78e4d8b0f9ef9df9fb01c5c828103deb9c57bcda90 Jul 28 00:12:52.495279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.495245 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r"] Jul 28 00:12:52.499104 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:12:52.499075 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5639af3a_9df3_4991_954d_edb789bf43b9.slice/crio-fdcf7ca88a33cdd093c9470f3c8f1485eb59e5a4e100c1be5f14f33b7dd3dc31 WatchSource:0}: Error finding container fdcf7ca88a33cdd093c9470f3c8f1485eb59e5a4e100c1be5f14f33b7dd3dc31: Status 404 returned error can't find the container with id fdcf7ca88a33cdd093c9470f3c8f1485eb59e5a4e100c1be5f14f33b7dd3dc31 Jul 28 00:12:52.526635 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.526613 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:12:52.545933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.545807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" event={"ID":"58be2baf-7afb-44d2-a659-e94fb5b94d8a","Type":"ContainerStarted","Data":"8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907"} Jul 28 00:12:52.545933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.545865 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:12:52.545933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.545888 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" event={"ID":"58be2baf-7afb-44d2-a659-e94fb5b94d8a","Type":"ContainerStarted","Data":"665d2be6ee6e3ce6d0105e78e4d8b0f9ef9df9fb01c5c828103deb9c57bcda90"} Jul 28 00:12:52.547563 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.547539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" event={"ID":"d9dad814-b19e-481f-b555-14c6733958ca","Type":"ContainerStarted","Data":"74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528"} Jul 28 00:12:52.547658 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.547585 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" event={"ID":"d9dad814-b19e-481f-b555-14c6733958ca","Type":"ContainerStarted","Data":"4f4d9d6a5142b665fc836e5054cd86cf49b39370b0a7a511bceb95c554b7ab62"} Jul 28 00:12:52.547658 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.547627 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:12:52.548888 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.548868 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" event={"ID":"5639af3a-9df3-4991-954d-edb789bf43b9","Type":"ContainerStarted","Data":"fdcf7ca88a33cdd093c9470f3c8f1485eb59e5a4e100c1be5f14f33b7dd3dc31"} Jul 28 00:12:52.597419 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.597233 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" podStartSLOduration=1.152673467 podStartE2EDuration="1.597218871s" totalImagesPullingTime="444.545404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:12:52.594780765 +0000 UTC m=+2172.518989667" watchObservedRunningTime="2026-07-28 00:12:52.597218871 +0000 UTC m=+2172.521427750" Jul 28 00:12:52.627948 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:52.627896 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" podStartSLOduration=1.6278764460000001 podStartE2EDuration="1.627876446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:12:52.626705379 +0000 UTC m=+2172.550914259" watchObservedRunningTime="2026-07-28 00:12:52.627876446 +0000 UTC m=+2172.552085323" Jul 28 00:12:53.555509 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:53.555456 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" event={"ID":"5639af3a-9df3-4991-954d-edb789bf43b9","Type":"ContainerStarted","Data":"bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935"} Jul 28 00:12:53.555933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:53.555683 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:12:53.571211 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:53.571164 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" podStartSLOduration=2.571147416 podStartE2EDuration="2.571147416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:12:53.570357087 +0000 UTC m=+2173.494565968" watchObservedRunningTime="2026-07-28 00:12:53.571147416 +0000 UTC m=+2173.495356296" Jul 28 00:12:55.512092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:55.512060 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:12:55.620649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:55.620614 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:56.518205 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:56.518172 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:12:58.565000 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:12:58.564973 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:13:03.558021 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:03.557987 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:13:04.370762 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:04.370728 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:04.471376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:04.471339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:04.566160 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:04.566131 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:13:05.922410 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.922379 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2"] Jul 28 00:13:05.925904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.925882 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:05.928737 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.928716 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f7352d\"/\"default-dockercfg-tw4ng\"" Jul 28 00:13:05.928874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.928758 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f7352d\"/\"test-model-f7352d-kserve-self-signed-certs\"" Jul 28 00:13:05.928874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.928761 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f7352d\"/\"kube-root-ca.crt\"" Jul 28 00:13:05.928874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.928837 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f7352d\"/\"openshift-service-ca.crt\"" Jul 28 00:13:05.935959 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.935932 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2"] Jul 28 00:13:05.946828 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:05.946801 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:06.094947 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.094913 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-dshm\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.095100 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.094969 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tls-certs\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.095100 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.094991 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-home\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.095100 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.095019 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55bh\" (UniqueName: \"kubernetes.io/projected/a4d7dc74-a60c-4504-8f75-8137859ea5c3-kube-api-access-w55bh\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.095100 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.095050 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-model-cache\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.095281 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.095110 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tmp-dir\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196477 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-dshm\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196477 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196454 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tls-certs\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196726 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-home\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196726 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196573 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55bh\" (UniqueName: \"kubernetes.io/projected/a4d7dc74-a60c-4504-8f75-8137859ea5c3-kube-api-access-w55bh\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196726 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196616 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-model-cache\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196726 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196641 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tmp-dir\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196986 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196963 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-home\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.196986 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.196980 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tmp-dir\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.197096 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.197044 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-model-cache\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.198745 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.198718 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-dshm\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.198997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.198977 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tls-certs\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.203628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.203602 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55bh\" (UniqueName: \"kubernetes.io/projected/a4d7dc74-a60c-4504-8f75-8137859ea5c3-kube-api-access-w55bh\") pod \"test-model-f7352d-kserve-78968867d7-z8bn2\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.236148 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.236120 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:06.373045 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.373021 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2"] Jul 28 00:13:06.374727 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:13:06.374690 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d7dc74_a60c_4504_8f75_8137859ea5c3.slice/crio-ab6d1ebdcb1181514b2eeac1fbd696bc231d37dffb435f60348b9cdb460d883f WatchSource:0}: Error finding container ab6d1ebdcb1181514b2eeac1fbd696bc231d37dffb435f60348b9cdb460d883f: Status 404 returned error can't find the container with id ab6d1ebdcb1181514b2eeac1fbd696bc231d37dffb435f60348b9cdb460d883f Jul 28 00:13:06.630335 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.630297 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" event={"ID":"a4d7dc74-a60c-4504-8f75-8137859ea5c3","Type":"ContainerStarted","Data":"7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f"} Jul 28 00:13:06.630335 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.630344 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" event={"ID":"a4d7dc74-a60c-4504-8f75-8137859ea5c3","Type":"ContainerStarted","Data":"ab6d1ebdcb1181514b2eeac1fbd696bc231d37dffb435f60348b9cdb460d883f"} Jul 28 00:13:06.645330 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:06.645276 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" podStartSLOduration=1.6452578679999998 podStartE2EDuration="1.645257868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:13:06.644342757 +0000 UTC m=+2186.568551650" watchObservedRunningTime="2026-07-28 00:13:06.645257868 +0000 UTC m=+2186.569466747" Jul 28 00:13:16.236882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:16.236846 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:16.236882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:16.236892 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:16.249153 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:16.249130 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:16.692339 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:16.692310 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:13:34.635557 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.635520 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk"] Jul 28 00:13:34.640562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.640542 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.643489 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.643467 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2ccf60\"/\"default-dockercfg-7xht5\"" Jul 28 00:13:34.643623 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.643525 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2ccf60\"/\"openshift-service-ca.crt\"" Jul 28 00:13:34.643623 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.643467 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2ccf60\"/\"kube-root-ca.crt\"" Jul 28 00:13:34.643623 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.643467 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2ccf60\"/\"test-model-2ccf60-kserve-self-signed-certs\"" Jul 28 00:13:34.648999 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.648268 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk"] Jul 28 00:13:34.653007 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.652982 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:34.755923 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.755890 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-dshm\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.756115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.755941 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tls-certs\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.756115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.756003 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2z89\" (UniqueName: \"kubernetes.io/projected/7f71457b-9ef9-4f30-a2d0-248393bb80dd-kube-api-access-j2z89\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.756115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.756056 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-home\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.756115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.756073 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-model-cache\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.756115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.756092 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tmp-dir\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857058 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857015 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-home\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857058 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857059 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-model-cache\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857315 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857085 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tmp-dir\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857315 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857156 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-dshm\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857315 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857300 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tls-certs\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857467 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857376 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z89\" (UniqueName: \"kubernetes.io/projected/7f71457b-9ef9-4f30-a2d0-248393bb80dd-kube-api-access-j2z89\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857557 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857541 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tmp-dir\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857608 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857562 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-model-cache\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.857672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.857648 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-home\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.859776 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.859754 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tls-certs\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.859883 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.859864 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-dshm\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.864257 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.864236 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z89\" (UniqueName: \"kubernetes.io/projected/7f71457b-9ef9-4f30-a2d0-248393bb80dd-kube-api-access-j2z89\") pod \"test-model-2ccf60-kserve-56d94b9757-sm8pk\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:34.951084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:34.950998 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:35.084697 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:35.084673 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk"] Jul 28 00:13:35.087204 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:13:35.087182 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f71457b_9ef9_4f30_a2d0_248393bb80dd.slice/crio-30472896f8fd793f78340cbe8d712c8ce8523f462bfe1fd695e2b513a2d15295 WatchSource:0}: Error finding container 30472896f8fd793f78340cbe8d712c8ce8523f462bfe1fd695e2b513a2d15295: Status 404 returned error can't find the container with id 30472896f8fd793f78340cbe8d712c8ce8523f462bfe1fd695e2b513a2d15295 Jul 28 00:13:35.780010 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:35.779970 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" event={"ID":"7f71457b-9ef9-4f30-a2d0-248393bb80dd","Type":"ContainerStarted","Data":"0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96"} Jul 28 00:13:35.780010 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:35.780018 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" event={"ID":"7f71457b-9ef9-4f30-a2d0-248393bb80dd","Type":"ContainerStarted","Data":"30472896f8fd793f78340cbe8d712c8ce8523f462bfe1fd695e2b513a2d15295"} Jul 28 00:13:35.799005 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:35.798959 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" podStartSLOduration=1.798943813 podStartE2EDuration="1.798943813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:13:35.797930269 +0000 UTC m=+2215.722139148" watchObservedRunningTime="2026-07-28 00:13:35.798943813 +0000 UTC m=+2215.723152690" Jul 28 00:13:40.983680 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:40.983645 2555 scope.go:117] "RemoveContainer" containerID="2334323ea29d4dbe9d311788e28a212e0b8a285b6349e3bece2e0eff902422df" Jul 28 00:13:44.951378 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:44.951333 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:44.951378 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:44.951387 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:44.964735 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:44.964707 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:13:45.839184 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:13:45.839148 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:14:15.738589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:15.738503 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2"] Jul 28 00:14:15.758031 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:15.758000 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:16.717008 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:16.716959 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk"] Jul 28 00:14:16.735764 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:16.735719 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:27.263116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.263083 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j"] Jul 28 00:14:27.263675 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.263536 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" podUID="587044f1-2d96-48d2-8517-8590d282b7c6" containerName="maas-api" containerID="cri-o://58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0" gracePeriod=30 Jul 28 00:14:27.293341 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.293302 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:27.309265 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:27.309238 2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587044f1_2d96_48d2_8517_8590d282b7c6.slice/crio-58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0.scope\": RecentStats: unable to find data in memory cache]" Jul 28 00:14:27.316191 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.316163 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf"] Jul 28 00:14:27.316441 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.316391 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" podUID="ace5567c-6162-4bea-af52-15f883238f42" containerName="payload-processing" containerID="cri-o://1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb" gracePeriod=30 Jul 28 00:14:27.326172 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.326141 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8"] Jul 28 00:14:27.326509 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.326469 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" podUID="f93a80ed-5422-42c3-a577-95824ddfd574" containerName="payload-pre-processing" containerID="cri-o://58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee" gracePeriod=30 Jul 28 00:14:27.358073 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.358042 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:27.412192 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.412156 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:27.521802 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.521488 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" podUID="587044f1-2d96-48d2-8517-8590d282b7c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.75:8443/health\": dial tcp 10.133.0.75:8443: connect: connection refused" Jul 28 00:14:27.618392 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.618366 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:14:27.638410 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.638139 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:14:27.643486 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.643462 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:14:27.758450 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758413 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace5567c-6162-4bea-af52-15f883238f42-plugins-config-volume\") pod \"ace5567c-6162-4bea-af52-15f883238f42\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " Jul 28 00:14:27.758450 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758451 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93a80ed-5422-42c3-a577-95824ddfd574-plugins-config-volume\") pod \"f93a80ed-5422-42c3-a577-95824ddfd574\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " Jul 28 00:14:27.758708 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758541 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls\") pod \"587044f1-2d96-48d2-8517-8590d282b7c6\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " Jul 28 00:14:27.758708 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758577 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtcns\" (UniqueName: \"kubernetes.io/projected/ace5567c-6162-4bea-af52-15f883238f42-kube-api-access-dtcns\") pod \"ace5567c-6162-4bea-af52-15f883238f42\" (UID: \"ace5567c-6162-4bea-af52-15f883238f42\") " Jul 28 00:14:27.758708 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758605 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x65g\" (UniqueName: \"kubernetes.io/projected/587044f1-2d96-48d2-8517-8590d282b7c6-kube-api-access-6x65g\") pod \"587044f1-2d96-48d2-8517-8590d282b7c6\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " Jul 28 00:14:27.758708 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758638 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587044f1-2d96-48d2-8517-8590d282b7c6-openshift-service-ca\") pod \"587044f1-2d96-48d2-8517-8590d282b7c6\" (UID: \"587044f1-2d96-48d2-8517-8590d282b7c6\") " Jul 28 00:14:27.758708 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758704 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw29t\" (UniqueName: \"kubernetes.io/projected/f93a80ed-5422-42c3-a577-95824ddfd574-kube-api-access-kw29t\") pod \"f93a80ed-5422-42c3-a577-95824ddfd574\" (UID: \"f93a80ed-5422-42c3-a577-95824ddfd574\") " Jul 28 00:14:27.759023 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.758904 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace5567c-6162-4bea-af52-15f883238f42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ace5567c-6162-4bea-af52-15f883238f42" (UID: "ace5567c-6162-4bea-af52-15f883238f42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:27.759098 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.759050 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93a80ed-5422-42c3-a577-95824ddfd574-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f93a80ed-5422-42c3-a577-95824ddfd574" (UID: "f93a80ed-5422-42c3-a577-95824ddfd574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:27.759151 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.759089 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587044f1-2d96-48d2-8517-8590d282b7c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "587044f1-2d96-48d2-8517-8590d282b7c6" (UID: "587044f1-2d96-48d2-8517-8590d282b7c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:27.759202 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.759186 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace5567c-6162-4bea-af52-15f883238f42-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:27.759258 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.759206 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93a80ed-5422-42c3-a577-95824ddfd574-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:27.759258 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.759221 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587044f1-2d96-48d2-8517-8590d282b7c6-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:27.760743 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.760714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "587044f1-2d96-48d2-8517-8590d282b7c6" (UID: "587044f1-2d96-48d2-8517-8590d282b7c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:14:27.760844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.760773 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587044f1-2d96-48d2-8517-8590d282b7c6-kube-api-access-6x65g" (OuterVolumeSpecName: "kube-api-access-6x65g") pod "587044f1-2d96-48d2-8517-8590d282b7c6" (UID: "587044f1-2d96-48d2-8517-8590d282b7c6"). InnerVolumeSpecName "kube-api-access-6x65g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:27.760885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.760858 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93a80ed-5422-42c3-a577-95824ddfd574-kube-api-access-kw29t" (OuterVolumeSpecName: "kube-api-access-kw29t") pod "f93a80ed-5422-42c3-a577-95824ddfd574" (UID: "f93a80ed-5422-42c3-a577-95824ddfd574"). InnerVolumeSpecName "kube-api-access-kw29t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:27.760885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.760854 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace5567c-6162-4bea-af52-15f883238f42-kube-api-access-dtcns" (OuterVolumeSpecName: "kube-api-access-dtcns") pod "ace5567c-6162-4bea-af52-15f883238f42" (UID: "ace5567c-6162-4bea-af52-15f883238f42"). InnerVolumeSpecName "kube-api-access-dtcns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:27.859915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.859819 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw29t\" (UniqueName: \"kubernetes.io/projected/f93a80ed-5422-42c3-a577-95824ddfd574-kube-api-access-kw29t\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:27.859915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.859853 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587044f1-2d96-48d2-8517-8590d282b7c6-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:27.859915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.859863 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtcns\" (UniqueName: \"kubernetes.io/projected/ace5567c-6162-4bea-af52-15f883238f42-kube-api-access-dtcns\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:27.859915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:27.859873 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x65g\" (UniqueName: \"kubernetes.io/projected/587044f1-2d96-48d2-8517-8590d282b7c6-kube-api-access-6x65g\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:28.038583 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.038541 2555 generic.go:356] "Generic (PLEG): container finished" podID="f93a80ed-5422-42c3-a577-95824ddfd574" containerID="58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee" exitCode=0 Jul 28 00:14:28.038775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.038591 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" Jul 28 00:14:28.038775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.038631 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" event={"ID":"f93a80ed-5422-42c3-a577-95824ddfd574","Type":"ContainerDied","Data":"58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee"} Jul 28 00:14:28.038775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.038687 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8" event={"ID":"f93a80ed-5422-42c3-a577-95824ddfd574","Type":"ContainerDied","Data":"8a9b55bb2bacfef43602624938a364940b0052d003659d53c7aa052e058efe9c"} Jul 28 00:14:28.038775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.038702 2555 scope.go:117] "RemoveContainer" containerID="58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee" Jul 28 00:14:28.039910 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.039827 2555 generic.go:356] "Generic (PLEG): container finished" podID="587044f1-2d96-48d2-8517-8590d282b7c6" containerID="58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0" exitCode=0 Jul 28 00:14:28.039910 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.039867 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" Jul 28 00:14:28.040018 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.039910 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" event={"ID":"587044f1-2d96-48d2-8517-8590d282b7c6","Type":"ContainerDied","Data":"58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0"} Jul 28 00:14:28.040018 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.039960 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j" event={"ID":"587044f1-2d96-48d2-8517-8590d282b7c6","Type":"ContainerDied","Data":"9fb278ec8b05725d4a1cdc77af702c43746aca69b3d39c978e6fddd911d91ca8"} Jul 28 00:14:28.041319 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.041280 2555 generic.go:356] "Generic (PLEG): container finished" podID="ace5567c-6162-4bea-af52-15f883238f42" containerID="1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb" exitCode=0 Jul 28 00:14:28.041319 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.041306 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" event={"ID":"ace5567c-6162-4bea-af52-15f883238f42","Type":"ContainerDied","Data":"1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb"} Jul 28 00:14:28.041443 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.041329 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" Jul 28 00:14:28.041443 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.041357 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf" event={"ID":"ace5567c-6162-4bea-af52-15f883238f42","Type":"ContainerDied","Data":"834fee2afea5ef68de497468b146212b25449fba65194e686df3c02d95dcee09"} Jul 28 00:14:28.049137 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.049117 2555 scope.go:117] "RemoveContainer" containerID="58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee" Jul 28 00:14:28.049425 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:28.049407 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee\": container with ID starting with 58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee not found: ID does not exist" containerID="58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee" Jul 28 00:14:28.049535 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.049437 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee"} err="failed to get container status \"58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee\": rpc error: code = NotFound desc = could not find container \"58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee\": container with ID starting with 58fc5f5318c8bc409a4b67e652a9f8e92565dea2918d3654655e4b5a155a2cee not found: ID does not exist" Jul 28 00:14:28.049535 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.049461 2555 scope.go:117] "RemoveContainer" containerID="58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0" Jul 28 00:14:28.057717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.057701 2555 scope.go:117] "RemoveContainer" containerID="58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0" Jul 28 00:14:28.057960 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:28.057939 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0\": container with ID starting with 58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0 not found: ID does not exist" containerID="58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0" Jul 28 00:14:28.058008 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.057967 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0"} err="failed to get container status \"58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0\": rpc error: code = NotFound desc = could not find container \"58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0\": container with ID starting with 58a70a00a335eebc538ee572c733431ca1f170d99a7e5c4848682e02b8a8afb0 not found: ID does not exist" Jul 28 00:14:28.058008 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.057984 2555 scope.go:117] "RemoveContainer" containerID="1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb" Jul 28 00:14:28.063043 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.063020 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8"] Jul 28 00:14:28.065964 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.065945 2555 scope.go:117] "RemoveContainer" containerID="1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb" Jul 28 00:14:28.066229 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:28.066210 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb\": container with ID starting with 1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb not found: ID does not exist" containerID="1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb" Jul 28 00:14:28.066291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.066236 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb"} err="failed to get container status \"1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb\": rpc error: code = NotFound desc = could not find container \"1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb\": container with ID starting with 1bb8eacf1d6f270228510032a5e3c9b8e3d1c032055acfce85141361e40649eb not found: ID does not exist" Jul 28 00:14:28.067025 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.067009 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f7352d-76c4545768-gckv8"] Jul 28 00:14:28.076079 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.076060 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j"] Jul 28 00:14:28.080649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.080631 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f7352d-7985555786-d7v4j"] Jul 28 00:14:28.088484 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.088464 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf"] Jul 28 00:14:28.093121 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.093101 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f7352d-6887d66748-nm2hf"] Jul 28 00:14:28.566123 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.566033 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587044f1-2d96-48d2-8517-8590d282b7c6" path="/var/lib/kubelet/pods/587044f1-2d96-48d2-8517-8590d282b7c6/volumes" Jul 28 00:14:28.566623 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.566533 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ace5567c-6162-4bea-af52-15f883238f42" path="/var/lib/kubelet/pods/ace5567c-6162-4bea-af52-15f883238f42/volumes" Jul 28 00:14:28.566928 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:28.566908 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f93a80ed-5422-42c3-a577-95824ddfd574" path="/var/lib/kubelet/pods/f93a80ed-5422-42c3-a577-95824ddfd574/volumes" Jul 28 00:14:30.758035 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:30.757999 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" podUID="a4d7dc74-a60c-4504-8f75-8137859ea5c3" containerName="main" containerID="cri-o://7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f" gracePeriod=45 Jul 28 00:14:31.021966 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.021906 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:14:31.062040 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.062004 2555 generic.go:356] "Generic (PLEG): container finished" podID="a4d7dc74-a60c-4504-8f75-8137859ea5c3" containerID="7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f" exitCode=0 Jul 28 00:14:31.062215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.062045 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" Jul 28 00:14:31.062215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.062062 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" event={"ID":"a4d7dc74-a60c-4504-8f75-8137859ea5c3","Type":"ContainerDied","Data":"7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f"} Jul 28 00:14:31.062215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.062096 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2" event={"ID":"a4d7dc74-a60c-4504-8f75-8137859ea5c3","Type":"ContainerDied","Data":"ab6d1ebdcb1181514b2eeac1fbd696bc231d37dffb435f60348b9cdb460d883f"} Jul 28 00:14:31.062215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.062116 2555 scope.go:117] "RemoveContainer" containerID="7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f" Jul 28 00:14:31.071726 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.071693 2555 scope.go:117] "RemoveContainer" containerID="7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f" Jul 28 00:14:31.071959 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:31.071937 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f\": container with ID starting with 7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f not found: ID does not exist" containerID="7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f" Jul 28 00:14:31.072043 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.071966 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f"} err="failed to get container status \"7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f\": rpc error: code = NotFound desc = could not find container \"7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f\": container with ID starting with 7ebe51568b353671e9535864b1cdd03761fc6cefcc5da6aa53dd999277913c0f not found: ID does not exist" Jul 28 00:14:31.188405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188363 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-model-cache\") pod \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " Jul 28 00:14:31.188600 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188427 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55bh\" (UniqueName: \"kubernetes.io/projected/a4d7dc74-a60c-4504-8f75-8137859ea5c3-kube-api-access-w55bh\") pod \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " Jul 28 00:14:31.188600 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188468 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tmp-dir\") pod \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " Jul 28 00:14:31.188600 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188538 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tls-certs\") pod \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " Jul 28 00:14:31.188723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188605 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-home\") pod \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " Jul 28 00:14:31.188723 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188651 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-dshm\") pod \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\" (UID: \"a4d7dc74-a60c-4504-8f75-8137859ea5c3\") " Jul 28 00:14:31.188799 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188729 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4d7dc74-a60c-4504-8f75-8137859ea5c3" (UID: "a4d7dc74-a60c-4504-8f75-8137859ea5c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:31.188799 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188747 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4d7dc74-a60c-4504-8f75-8137859ea5c3" (UID: "a4d7dc74-a60c-4504-8f75-8137859ea5c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:31.188908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188888 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-home" (OuterVolumeSpecName: "home") pod "a4d7dc74-a60c-4504-8f75-8137859ea5c3" (UID: "a4d7dc74-a60c-4504-8f75-8137859ea5c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:31.188969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188953 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:31.189007 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188974 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:31.189007 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.188986 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:31.190701 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.190680 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4d7dc74-a60c-4504-8f75-8137859ea5c3" (UID: "a4d7dc74-a60c-4504-8f75-8137859ea5c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:14:31.190765 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.190716 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-dshm" (OuterVolumeSpecName: "dshm") pod "a4d7dc74-a60c-4504-8f75-8137859ea5c3" (UID: "a4d7dc74-a60c-4504-8f75-8137859ea5c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:31.190800 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.190769 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d7dc74-a60c-4504-8f75-8137859ea5c3-kube-api-access-w55bh" (OuterVolumeSpecName: "kube-api-access-w55bh") pod "a4d7dc74-a60c-4504-8f75-8137859ea5c3" (UID: "a4d7dc74-a60c-4504-8f75-8137859ea5c3"). InnerVolumeSpecName "kube-api-access-w55bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:31.289649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.289550 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d7dc74-a60c-4504-8f75-8137859ea5c3-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:31.289649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.289585 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55bh\" (UniqueName: \"kubernetes.io/projected/a4d7dc74-a60c-4504-8f75-8137859ea5c3-kube-api-access-w55bh\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:31.289649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.289599 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7dc74-a60c-4504-8f75-8137859ea5c3-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:31.383553 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.383519 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2"] Jul 28 00:14:31.392087 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.392046 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f7352d/test-model-f7352d-kserve-78968867d7-z8bn2"] Jul 28 00:14:31.738632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.738535 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" podUID="7f71457b-9ef9-4f30-a2d0-248393bb80dd" containerName="main" containerID="cri-o://0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96" gracePeriod=45 Jul 28 00:14:31.952784 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.952753 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:31.995980 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:31.995952 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:14:32.069403 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.069369 2555 generic.go:356] "Generic (PLEG): container finished" podID="7f71457b-9ef9-4f30-a2d0-248393bb80dd" containerID="0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96" exitCode=0 Jul 28 00:14:32.069588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.069413 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" Jul 28 00:14:32.069588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.069456 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" event={"ID":"7f71457b-9ef9-4f30-a2d0-248393bb80dd","Type":"ContainerDied","Data":"0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96"} Jul 28 00:14:32.069588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.069523 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk" event={"ID":"7f71457b-9ef9-4f30-a2d0-248393bb80dd","Type":"ContainerDied","Data":"30472896f8fd793f78340cbe8d712c8ce8523f462bfe1fd695e2b513a2d15295"} Jul 28 00:14:32.069588 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.069545 2555 scope.go:117] "RemoveContainer" containerID="0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96" Jul 28 00:14:32.077959 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.077943 2555 scope.go:117] "RemoveContainer" containerID="0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96" Jul 28 00:14:32.078232 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:32.078215 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96\": container with ID starting with 0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96 not found: ID does not exist" containerID="0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96" Jul 28 00:14:32.078279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.078242 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96"} err="failed to get container status \"0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96\": rpc error: code = NotFound desc = could not find container \"0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96\": container with ID starting with 0fc534f6c36ed6dfe0f1736b9f87f783726bc9bd966c47f34bfdec45e0545a96 not found: ID does not exist" Jul 28 00:14:32.097557 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097529 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tls-certs\") pod \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " Jul 28 00:14:32.097659 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097584 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2z89\" (UniqueName: \"kubernetes.io/projected/7f71457b-9ef9-4f30-a2d0-248393bb80dd-kube-api-access-j2z89\") pod \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " Jul 28 00:14:32.097659 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097628 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tmp-dir\") pod \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " Jul 28 00:14:32.097734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097680 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-home\") pod \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " Jul 28 00:14:32.097734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097708 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-model-cache\") pod \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " Jul 28 00:14:32.097831 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097786 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-dshm\") pod \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\" (UID: \"7f71457b-9ef9-4f30-a2d0-248393bb80dd\") " Jul 28 00:14:32.097948 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.097921 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f71457b-9ef9-4f30-a2d0-248393bb80dd" (UID: "7f71457b-9ef9-4f30-a2d0-248393bb80dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:32.098055 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.098023 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-home" (OuterVolumeSpecName: "home") pod "7f71457b-9ef9-4f30-a2d0-248393bb80dd" (UID: "7f71457b-9ef9-4f30-a2d0-248393bb80dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:32.098124 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.098047 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f71457b-9ef9-4f30-a2d0-248393bb80dd" (UID: "7f71457b-9ef9-4f30-a2d0-248393bb80dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:32.098124 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.098085 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:32.099711 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.099690 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-dshm" (OuterVolumeSpecName: "dshm") pod "7f71457b-9ef9-4f30-a2d0-248393bb80dd" (UID: "7f71457b-9ef9-4f30-a2d0-248393bb80dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:14:32.099798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.099783 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f71457b-9ef9-4f30-a2d0-248393bb80dd" (UID: "7f71457b-9ef9-4f30-a2d0-248393bb80dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:14:32.099863 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.099839 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f71457b-9ef9-4f30-a2d0-248393bb80dd-kube-api-access-j2z89" (OuterVolumeSpecName: "kube-api-access-j2z89") pod "7f71457b-9ef9-4f30-a2d0-248393bb80dd" (UID: "7f71457b-9ef9-4f30-a2d0-248393bb80dd"). InnerVolumeSpecName "kube-api-access-j2z89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:32.199142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.199109 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:32.199142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.199140 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f71457b-9ef9-4f30-a2d0-248393bb80dd-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:32.199142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.199157 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2z89\" (UniqueName: \"kubernetes.io/projected/7f71457b-9ef9-4f30-a2d0-248393bb80dd-kube-api-access-j2z89\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:32.199382 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.199171 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:32.199382 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.199183 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f71457b-9ef9-4f30-a2d0-248393bb80dd-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:32.399505 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.399471 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:32.409116 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.409092 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk"] Jul 28 00:14:32.415260 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.415237 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2ccf60/test-model-2ccf60-kserve-56d94b9757-sm8pk"] Jul 28 00:14:32.566166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.566132 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f71457b-9ef9-4f30-a2d0-248393bb80dd" path="/var/lib/kubelet/pods/7f71457b-9ef9-4f30-a2d0-248393bb80dd/volumes" Jul 28 00:14:32.566561 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:32.566545 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d7dc74-a60c-4504-8f75-8137859ea5c3" path="/var/lib/kubelet/pods/a4d7dc74-a60c-4504-8f75-8137859ea5c3/volumes" Jul 28 00:14:38.169737 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.169702 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:38.228391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.228355 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:38.497668 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.497631 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:38.696748 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.696693 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:38.809752 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.809677 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw"] Jul 28 00:14:38.810067 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810055 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d7dc74-a60c-4504-8f75-8137859ea5c3" containerName="main" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810069 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d7dc74-a60c-4504-8f75-8137859ea5c3" containerName="main" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810080 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace5567c-6162-4bea-af52-15f883238f42" containerName="payload-processing" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810085 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace5567c-6162-4bea-af52-15f883238f42" containerName="payload-processing" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810094 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587044f1-2d96-48d2-8517-8590d282b7c6" containerName="maas-api" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810099 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="587044f1-2d96-48d2-8517-8590d282b7c6" containerName="maas-api" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810109 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93a80ed-5422-42c3-a577-95824ddfd574" containerName="payload-pre-processing" Jul 28 00:14:38.810112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810115 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93a80ed-5422-42c3-a577-95824ddfd574" containerName="payload-pre-processing" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810127 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f71457b-9ef9-4f30-a2d0-248393bb80dd" containerName="main" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810133 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f71457b-9ef9-4f30-a2d0-248393bb80dd" containerName="main" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810195 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f93a80ed-5422-42c3-a577-95824ddfd574" containerName="payload-pre-processing" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810203 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d7dc74-a60c-4504-8f75-8137859ea5c3" containerName="main" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810210 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f71457b-9ef9-4f30-a2d0-248393bb80dd" containerName="main" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810217 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="587044f1-2d96-48d2-8517-8590d282b7c6" containerName="maas-api" Jul 28 00:14:38.810326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.810223 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace5567c-6162-4bea-af52-15f883238f42" containerName="payload-processing" Jul 28 00:14:38.813400 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.813383 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:38.815590 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.815567 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:14:38.819378 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.819353 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw"] Jul 28 00:14:38.958811 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:38.958776 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a272fc38-5b9d-4baa-80c9-bf613b3a6686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw\" (UID: \"a272fc38-5b9d-4baa-80c9-bf613b3a6686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:39.059851 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:39.059763 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a272fc38-5b9d-4baa-80c9-bf613b3a6686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw\" (UID: \"a272fc38-5b9d-4baa-80c9-bf613b3a6686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:39.060409 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:39.060381 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a272fc38-5b9d-4baa-80c9-bf613b3a6686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw\" (UID: \"a272fc38-5b9d-4baa-80c9-bf613b3a6686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:39.125635 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:39.125610 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:39.272942 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:39.272916 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw"] Jul 28 00:14:39.274760 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:14:39.274736 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda272fc38_5b9d_4baa_80c9_bf613b3a6686.slice/crio-6cb5676e1e0e30b583755ebc7d0de430f31defa6c25ec9ff8dd5429f23058c15 WatchSource:0}: Error finding container 6cb5676e1e0e30b583755ebc7d0de430f31defa6c25ec9ff8dd5429f23058c15: Status 404 returned error can't find the container with id 6cb5676e1e0e30b583755ebc7d0de430f31defa6c25ec9ff8dd5429f23058c15 Jul 28 00:14:40.112742 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:40.112709 2555 generic.go:356] "Generic (PLEG): container finished" podID="a272fc38-5b9d-4baa-80c9-bf613b3a6686" containerID="bdb05fff9b9f97791bd8d794b48b0540909c2592f02b62e5283c1af2e1eed462" exitCode=0 Jul 28 00:14:40.112893 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:40.112789 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" event={"ID":"a272fc38-5b9d-4baa-80c9-bf613b3a6686","Type":"ContainerDied","Data":"bdb05fff9b9f97791bd8d794b48b0540909c2592f02b62e5283c1af2e1eed462"} Jul 28 00:14:40.112893 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:40.112842 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" event={"ID":"a272fc38-5b9d-4baa-80c9-bf613b3a6686","Type":"ContainerStarted","Data":"6cb5676e1e0e30b583755ebc7d0de430f31defa6c25ec9ff8dd5429f23058c15"} Jul 28 00:14:41.249641 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:41.249614 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:41.380266 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:41.380168 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a272fc38-5b9d-4baa-80c9-bf613b3a6686-maas-api-service-ca\") pod \"a272fc38-5b9d-4baa-80c9-bf613b3a6686\" (UID: \"a272fc38-5b9d-4baa-80c9-bf613b3a6686\") " Jul 28 00:14:41.380562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:41.380539 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a272fc38-5b9d-4baa-80c9-bf613b3a6686-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a272fc38-5b9d-4baa-80c9-bf613b3a6686" (UID: "a272fc38-5b9d-4baa-80c9-bf613b3a6686"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:41.480989 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:41.480956 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a272fc38-5b9d-4baa-80c9-bf613b3a6686-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:42.129159 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:42.129122 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" event={"ID":"a272fc38-5b9d-4baa-80c9-bf613b3a6686","Type":"ContainerDied","Data":"6cb5676e1e0e30b583755ebc7d0de430f31defa6c25ec9ff8dd5429f23058c15"} Jul 28 00:14:42.129317 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:42.129176 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cb5676e1e0e30b583755ebc7d0de430f31defa6c25ec9ff8dd5429f23058c15" Jul 28 00:14:42.129317 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:42.129140 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw" Jul 28 00:14:49.205332 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.205298 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb"] Jul 28 00:14:49.205817 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.205554 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" podUID="d9dad814-b19e-481f-b555-14c6733958ca" containerName="maas-api" containerID="cri-o://74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528" gracePeriod=30 Jul 28 00:14:49.221543 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.221512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:49.246724 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.246689 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22"] Jul 28 00:14:49.246955 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.246929 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" podUID="58be2baf-7afb-44d2-a659-e94fb5b94d8a" containerName="payload-processing" containerID="cri-o://8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907" gracePeriod=30 Jul 28 00:14:49.253278 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.253251 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r"] Jul 28 00:14:49.253657 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.253474 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" podUID="5639af3a-9df3-4991-954d-edb789bf43b9" containerName="payload-pre-processing" containerID="cri-o://bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935" gracePeriod=30 Jul 28 00:14:49.285288 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.285249 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:49.345289 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.345254 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:49.545245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.545220 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:14:49.548137 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.548112 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:14:49.561673 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.561650 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:14:49.649765 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649731 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dad814-b19e-481f-b555-14c6733958ca-openshift-service-ca\") pod \"d9dad814-b19e-481f-b555-14c6733958ca\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " Jul 28 00:14:49.649932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649802 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmp9w\" (UniqueName: \"kubernetes.io/projected/58be2baf-7afb-44d2-a659-e94fb5b94d8a-kube-api-access-kmp9w\") pod \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " Jul 28 00:14:49.649932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649846 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9dad814-b19e-481f-b555-14c6733958ca-maas-api-tls\") pod \"d9dad814-b19e-481f-b555-14c6733958ca\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " Jul 28 00:14:49.649932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649864 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5639af3a-9df3-4991-954d-edb789bf43b9-plugins-config-volume\") pod \"5639af3a-9df3-4991-954d-edb789bf43b9\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " Jul 28 00:14:49.649932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649905 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58be2baf-7afb-44d2-a659-e94fb5b94d8a-plugins-config-volume\") pod \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\" (UID: \"58be2baf-7afb-44d2-a659-e94fb5b94d8a\") " Jul 28 00:14:49.649932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649920 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dhrx\" (UniqueName: \"kubernetes.io/projected/d9dad814-b19e-481f-b555-14c6733958ca-kube-api-access-9dhrx\") pod \"d9dad814-b19e-481f-b555-14c6733958ca\" (UID: \"d9dad814-b19e-481f-b555-14c6733958ca\") " Jul 28 00:14:49.650208 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.649941 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tgvf\" (UniqueName: \"kubernetes.io/projected/5639af3a-9df3-4991-954d-edb789bf43b9-kube-api-access-2tgvf\") pod \"5639af3a-9df3-4991-954d-edb789bf43b9\" (UID: \"5639af3a-9df3-4991-954d-edb789bf43b9\") " Jul 28 00:14:49.650266 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.650208 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dad814-b19e-481f-b555-14c6733958ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9dad814-b19e-481f-b555-14c6733958ca" (UID: "d9dad814-b19e-481f-b555-14c6733958ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:49.650329 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.650279 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5639af3a-9df3-4991-954d-edb789bf43b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5639af3a-9df3-4991-954d-edb789bf43b9" (UID: "5639af3a-9df3-4991-954d-edb789bf43b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:49.650329 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.650285 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58be2baf-7afb-44d2-a659-e94fb5b94d8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58be2baf-7afb-44d2-a659-e94fb5b94d8a" (UID: "58be2baf-7afb-44d2-a659-e94fb5b94d8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:14:49.652180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.652149 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dad814-b19e-481f-b555-14c6733958ca-kube-api-access-9dhrx" (OuterVolumeSpecName: "kube-api-access-9dhrx") pod "d9dad814-b19e-481f-b555-14c6733958ca" (UID: "d9dad814-b19e-481f-b555-14c6733958ca"). InnerVolumeSpecName "kube-api-access-9dhrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:49.652180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.652172 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5639af3a-9df3-4991-954d-edb789bf43b9-kube-api-access-2tgvf" (OuterVolumeSpecName: "kube-api-access-2tgvf") pod "5639af3a-9df3-4991-954d-edb789bf43b9" (UID: "5639af3a-9df3-4991-954d-edb789bf43b9"). InnerVolumeSpecName "kube-api-access-2tgvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:49.652325 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.652191 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58be2baf-7afb-44d2-a659-e94fb5b94d8a-kube-api-access-kmp9w" (OuterVolumeSpecName: "kube-api-access-kmp9w") pod "58be2baf-7afb-44d2-a659-e94fb5b94d8a" (UID: "58be2baf-7afb-44d2-a659-e94fb5b94d8a"). InnerVolumeSpecName "kube-api-access-kmp9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:14:49.652325 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.652227 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dad814-b19e-481f-b555-14c6733958ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9dad814-b19e-481f-b555-14c6733958ca" (UID: "d9dad814-b19e-481f-b555-14c6733958ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:14:49.751177 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751148 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmp9w\" (UniqueName: \"kubernetes.io/projected/58be2baf-7afb-44d2-a659-e94fb5b94d8a-kube-api-access-kmp9w\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:49.751177 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751175 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9dad814-b19e-481f-b555-14c6733958ca-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:49.751177 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751184 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5639af3a-9df3-4991-954d-edb789bf43b9-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:49.751425 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751193 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58be2baf-7afb-44d2-a659-e94fb5b94d8a-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:49.751425 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751221 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dhrx\" (UniqueName: \"kubernetes.io/projected/d9dad814-b19e-481f-b555-14c6733958ca-kube-api-access-9dhrx\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:49.751425 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751232 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tgvf\" (UniqueName: \"kubernetes.io/projected/5639af3a-9df3-4991-954d-edb789bf43b9-kube-api-access-2tgvf\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:49.751425 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:49.751242 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9dad814-b19e-481f-b555-14c6733958ca-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:14:50.171597 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.171477 2555 generic.go:356] "Generic (PLEG): container finished" podID="d9dad814-b19e-481f-b555-14c6733958ca" containerID="74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528" exitCode=0 Jul 28 00:14:50.171597 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.171541 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" Jul 28 00:14:50.171597 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.171566 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" event={"ID":"d9dad814-b19e-481f-b555-14c6733958ca","Type":"ContainerDied","Data":"74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528"} Jul 28 00:14:50.171881 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.171616 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb" event={"ID":"d9dad814-b19e-481f-b555-14c6733958ca","Type":"ContainerDied","Data":"4f4d9d6a5142b665fc836e5054cd86cf49b39370b0a7a511bceb95c554b7ab62"} Jul 28 00:14:50.171881 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.171666 2555 scope.go:117] "RemoveContainer" containerID="74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528" Jul 28 00:14:50.173011 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.172989 2555 generic.go:356] "Generic (PLEG): container finished" podID="5639af3a-9df3-4991-954d-edb789bf43b9" containerID="bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935" exitCode=0 Jul 28 00:14:50.173118 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.173029 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" Jul 28 00:14:50.173118 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.173068 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" event={"ID":"5639af3a-9df3-4991-954d-edb789bf43b9","Type":"ContainerDied","Data":"bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935"} Jul 28 00:14:50.173234 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.173131 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r" event={"ID":"5639af3a-9df3-4991-954d-edb789bf43b9","Type":"ContainerDied","Data":"fdcf7ca88a33cdd093c9470f3c8f1485eb59e5a4e100c1be5f14f33b7dd3dc31"} Jul 28 00:14:50.174763 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.174560 2555 generic.go:356] "Generic (PLEG): container finished" podID="58be2baf-7afb-44d2-a659-e94fb5b94d8a" containerID="8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907" exitCode=0 Jul 28 00:14:50.174763 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.174582 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" event={"ID":"58be2baf-7afb-44d2-a659-e94fb5b94d8a","Type":"ContainerDied","Data":"8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907"} Jul 28 00:14:50.174763 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.174615 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" Jul 28 00:14:50.174763 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.174628 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22" event={"ID":"58be2baf-7afb-44d2-a659-e94fb5b94d8a","Type":"ContainerDied","Data":"665d2be6ee6e3ce6d0105e78e4d8b0f9ef9df9fb01c5c828103deb9c57bcda90"} Jul 28 00:14:50.182402 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.182299 2555 scope.go:117] "RemoveContainer" containerID="74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528" Jul 28 00:14:50.182722 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:50.182692 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528\": container with ID starting with 74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528 not found: ID does not exist" containerID="74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528" Jul 28 00:14:50.182832 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.182731 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528"} err="failed to get container status \"74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528\": rpc error: code = NotFound desc = could not find container \"74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528\": container with ID starting with 74d5235376ea48dfbed97a9ac3211cafbf627dd5a2f0b3da9450d7bf9d31a528 not found: ID does not exist" Jul 28 00:14:50.182832 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.182753 2555 scope.go:117] "RemoveContainer" containerID="bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935" Jul 28 00:14:50.190790 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.190775 2555 scope.go:117] "RemoveContainer" containerID="bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935" Jul 28 00:14:50.191030 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:50.191012 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935\": container with ID starting with bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935 not found: ID does not exist" containerID="bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935" Jul 28 00:14:50.191086 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.191036 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935"} err="failed to get container status \"bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935\": rpc error: code = NotFound desc = could not find container \"bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935\": container with ID starting with bccaf6c79c98f1f54702842eef67f090b8f867a08535e5f9c80c5a67f3552935 not found: ID does not exist" Jul 28 00:14:50.191086 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.191050 2555 scope.go:117] "RemoveContainer" containerID="8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907" Jul 28 00:14:50.198061 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.198040 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb"] Jul 28 00:14:50.199128 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.199054 2555 scope.go:117] "RemoveContainer" containerID="8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907" Jul 28 00:14:50.200264 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:14:50.199768 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907\": container with ID starting with 8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907 not found: ID does not exist" containerID="8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907" Jul 28 00:14:50.200264 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.199804 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907"} err="failed to get container status \"8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907\": rpc error: code = NotFound desc = could not find container \"8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907\": container with ID starting with 8949be9b7598d3e4b0075200080af47539607a0616b94c1ada41019a2a63e907 not found: ID does not exist" Jul 28 00:14:50.201882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.201856 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ccf60-75df949c84-skspb"] Jul 28 00:14:50.210900 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.210879 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22"] Jul 28 00:14:50.214281 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.214259 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ccf60-76dbb946bd-l7s22"] Jul 28 00:14:50.223790 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.223769 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r"] Jul 28 00:14:50.225403 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.225385 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ccf60-7848997d5b-z294r"] Jul 28 00:14:50.564546 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.564515 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5639af3a-9df3-4991-954d-edb789bf43b9" path="/var/lib/kubelet/pods/5639af3a-9df3-4991-954d-edb789bf43b9/volumes" Jul 28 00:14:50.564872 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.564860 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58be2baf-7afb-44d2-a659-e94fb5b94d8a" path="/var/lib/kubelet/pods/58be2baf-7afb-44d2-a659-e94fb5b94d8a/volumes" Jul 28 00:14:50.565202 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:50.565190 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9dad814-b19e-481f-b555-14c6733958ca" path="/var/lib/kubelet/pods/d9dad814-b19e-481f-b555-14c6733958ca/volumes" Jul 28 00:14:53.877672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:53.877628 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw"] Jul 28 00:14:53.880180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:53.880152 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ccf60-x9nqw"] Jul 28 00:14:53.882511 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:53.882463 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:54.565306 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:54.565272 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a272fc38-5b9d-4baa-80c9-bf613b3a6686" path="/var/lib/kubelet/pods/a272fc38-5b9d-4baa-80c9-bf613b3a6686/volumes" Jul 28 00:14:55.336037 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:55.336003 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:14:59.957130 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:14:59.957095 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:00.016650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.016609 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:00.155293 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155258 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx"] Jul 28 00:15:00.155673 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155660 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58be2baf-7afb-44d2-a659-e94fb5b94d8a" containerName="payload-processing" Jul 28 00:15:00.155717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155675 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="58be2baf-7afb-44d2-a659-e94fb5b94d8a" containerName="payload-processing" Jul 28 00:15:00.155717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155683 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a272fc38-5b9d-4baa-80c9-bf613b3a6686" containerName="revoke-keys" Jul 28 00:15:00.155717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155689 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a272fc38-5b9d-4baa-80c9-bf613b3a6686" containerName="revoke-keys" Jul 28 00:15:00.155717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155710 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dad814-b19e-481f-b555-14c6733958ca" containerName="maas-api" Jul 28 00:15:00.155717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155716 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dad814-b19e-481f-b555-14c6733958ca" containerName="maas-api" Jul 28 00:15:00.155878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155722 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5639af3a-9df3-4991-954d-edb789bf43b9" containerName="payload-pre-processing" Jul 28 00:15:00.155878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155728 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5639af3a-9df3-4991-954d-edb789bf43b9" containerName="payload-pre-processing" Jul 28 00:15:00.155878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155781 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dad814-b19e-481f-b555-14c6733958ca" containerName="maas-api" Jul 28 00:15:00.155878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155789 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="58be2baf-7afb-44d2-a659-e94fb5b94d8a" containerName="payload-processing" Jul 28 00:15:00.155878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155795 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5639af3a-9df3-4991-954d-edb789bf43b9" containerName="payload-pre-processing" Jul 28 00:15:00.155878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.155803 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a272fc38-5b9d-4baa-80c9-bf613b3a6686" containerName="revoke-keys" Jul 28 00:15:00.160425 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.160402 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" Jul 28 00:15:00.163195 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.163172 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:15:00.169999 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.169977 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" Jul 28 00:15:00.206308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.206274 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx"] Jul 28 00:15:00.327231 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.327189 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:00.524797 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.524765 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:00.532882 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.532816 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx"] Jul 28 00:15:00.534985 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:00.534960 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07aa5c98_acbc_4786_a8bf_abcac684a963.slice/crio-93b0e83f71ec6facaf829b9d8bbe9a82766455263ebcab1de6588fae59118fe4 WatchSource:0}: Error finding container 93b0e83f71ec6facaf829b9d8bbe9a82766455263ebcab1de6588fae59118fe4: Status 404 returned error can't find the container with id 93b0e83f71ec6facaf829b9d8bbe9a82766455263ebcab1de6588fae59118fe4 Jul 28 00:15:00.881996 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:00.881957 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:01.200004 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:01.199895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:01.234405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:01.234365 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" event={"ID":"07aa5c98-acbc-4786-a8bf-abcac684a963","Type":"ContainerStarted","Data":"2b30c228e91fb3eeae1836c27eac53a87b684434197a5355668abe267436dcc2"} Jul 28 00:15:01.234622 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:01.234419 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" event={"ID":"07aa5c98-acbc-4786-a8bf-abcac684a963","Type":"ContainerStarted","Data":"93b0e83f71ec6facaf829b9d8bbe9a82766455263ebcab1de6588fae59118fe4"} Jul 28 00:15:01.247512 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:01.247443 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" podStartSLOduration=1.247428154 podStartE2EDuration="1.247428154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:01.246961481 +0000 UTC m=+2301.171170360" watchObservedRunningTime="2026-07-28 00:15:01.247428154 +0000 UTC m=+2301.171637032" Jul 28 00:15:01.688341 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:01.688302 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:02.241286 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:02.241249 2555 generic.go:356] "Generic (PLEG): container finished" podID="07aa5c98-acbc-4786-a8bf-abcac684a963" containerID="2b30c228e91fb3eeae1836c27eac53a87b684434197a5355668abe267436dcc2" exitCode=0 Jul 28 00:15:02.241775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:02.241332 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" event={"ID":"07aa5c98-acbc-4786-a8bf-abcac684a963","Type":"ContainerDied","Data":"2b30c228e91fb3eeae1836c27eac53a87b684434197a5355668abe267436dcc2"} Jul 28 00:15:03.385631 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:03.385598 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" Jul 28 00:15:03.876754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:03.876707 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:03.977261 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:03.977219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:04.030831 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.030796 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7"] Jul 28 00:15:04.031263 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.031244 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07aa5c98-acbc-4786-a8bf-abcac684a963" containerName="cleanup" Jul 28 00:15:04.031263 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.031263 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07aa5c98-acbc-4786-a8bf-abcac684a963" containerName="cleanup" Jul 28 00:15:04.031458 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.031351 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07aa5c98-acbc-4786-a8bf-abcac684a963" containerName="cleanup" Jul 28 00:15:04.034724 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.034701 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.036824 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.036798 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cd61a5-serving-cert\"" Jul 28 00:15:04.045616 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.045592 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7"] Jul 28 00:15:04.065353 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.065315 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:04.073414 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.073382 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxgl\" (UniqueName: \"kubernetes.io/projected/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-kube-api-access-8sxgl\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.073560 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.073462 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.073560 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.073525 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.174485 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.174389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.174485 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.174447 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.174724 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.174545 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxgl\" (UniqueName: \"kubernetes.io/projected/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-kube-api-access-8sxgl\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.174724 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:15:04.174615 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-serving-cert: secret "maas-api-e2e-api-a-cd61a5-serving-cert" not found Jul 28 00:15:04.174724 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:15:04.174692 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls podName:010d41f1-9c5d-4da2-8222-fcc4c4848c5f nodeName:}" failed. No retries permitted until 2026-07-28 00:15:04.674676509 +0000 UTC m=+2304.598885365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls") pod "maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" (UID: "010d41f1-9c5d-4da2-8222-fcc4c4848c5f") : secret "maas-api-e2e-api-a-cd61a5-serving-cert" not found Jul 28 00:15:04.175236 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.175217 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.183792 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.183760 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxgl\" (UniqueName: \"kubernetes.io/projected/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-kube-api-access-8sxgl\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.222961 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.222919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:04.252987 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.252959 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" Jul 28 00:15:04.253161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.252986 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753295-t8htx" event={"ID":"07aa5c98-acbc-4786-a8bf-abcac684a963","Type":"ContainerDied","Data":"93b0e83f71ec6facaf829b9d8bbe9a82766455263ebcab1de6588fae59118fe4"} Jul 28 00:15:04.253161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.253028 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b0e83f71ec6facaf829b9d8bbe9a82766455263ebcab1de6588fae59118fe4" Jul 28 00:15:04.538174 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.538134 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:04.643153 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.643115 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs"] Jul 28 00:15:04.646615 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.646597 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.648787 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.648758 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd61a5\"" Jul 28 00:15:04.648922 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.648758 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd61a5-dockercfg-rhmh2\"" Jul 28 00:15:04.654354 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.654025 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs"] Jul 28 00:15:04.654354 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.654332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:04.679900 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.679861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.680805 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.680777 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq"] Jul 28 00:15:04.682489 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.682457 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls\") pod \"maas-api-e2e-api-a-cd61a5-5675595558-w6dr7\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.684245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.684227 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.698705 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.698670 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq"] Jul 28 00:15:04.780837 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.780797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8pv\" (UniqueName: \"kubernetes.io/projected/4be04558-0ee7-4188-8288-1d8809469a17-kube-api-access-fk8pv\") pod \"payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.781038 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.780849 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llwn\" (UniqueName: \"kubernetes.io/projected/4526dd79-f188-44e5-a245-7238c95d29fa-kube-api-access-7llwn\") pod \"payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.781038 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.780919 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be04558-0ee7-4188-8288-1d8809469a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.781038 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.780965 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526dd79-f188-44e5-a245-7238c95d29fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.882428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.882328 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8pv\" (UniqueName: \"kubernetes.io/projected/4be04558-0ee7-4188-8288-1d8809469a17-kube-api-access-fk8pv\") pod \"payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.882428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.882372 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llwn\" (UniqueName: \"kubernetes.io/projected/4526dd79-f188-44e5-a245-7238c95d29fa-kube-api-access-7llwn\") pod \"payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.882428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.882397 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be04558-0ee7-4188-8288-1d8809469a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.882754 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.882433 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526dd79-f188-44e5-a245-7238c95d29fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.883228 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.883205 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526dd79-f188-44e5-a245-7238c95d29fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.883344 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.883318 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be04558-0ee7-4188-8288-1d8809469a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.890132 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.890098 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8pv\" (UniqueName: \"kubernetes.io/projected/4be04558-0ee7-4188-8288-1d8809469a17-kube-api-access-fk8pv\") pod \"payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:04.890256 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.890181 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llwn\" (UniqueName: \"kubernetes.io/projected/4526dd79-f188-44e5-a245-7238c95d29fa-kube-api-access-7llwn\") pod \"payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:04.946634 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.946605 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:04.958374 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:04.958353 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:05.000555 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.000521 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:05.113409 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.113306 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7"] Jul 28 00:15:05.116995 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:05.115505 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010d41f1_9c5d_4da2_8222_fcc4c4848c5f.slice/crio-bafb53421f40c3de610625de45791f65d1e82e32ee825da010f2467596b4e8fb WatchSource:0}: Error finding container bafb53421f40c3de610625de45791f65d1e82e32ee825da010f2467596b4e8fb: Status 404 returned error can't find the container with id bafb53421f40c3de610625de45791f65d1e82e32ee825da010f2467596b4e8fb Jul 28 00:15:05.117935 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.117916 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:15:05.136926 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.136825 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs"] Jul 28 00:15:05.140104 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:05.140064 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be04558_0ee7_4188_8288_1d8809469a17.slice/crio-a9c12e5582f190dbdfe8d057b65f0b13ede25be00162c271ef1fdec83ecad385 WatchSource:0}: Error finding container a9c12e5582f190dbdfe8d057b65f0b13ede25be00162c271ef1fdec83ecad385: Status 404 returned error can't find the container with id a9c12e5582f190dbdfe8d057b65f0b13ede25be00162c271ef1fdec83ecad385 Jul 28 00:15:05.170970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.170947 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq"] Jul 28 00:15:05.175678 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:05.175646 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4526dd79_f188_44e5_a245_7238c95d29fa.slice/crio-f375b025680e93235f06dd00da88592d8d65bd7468226fad5f8eea84545ed3da WatchSource:0}: Error finding container f375b025680e93235f06dd00da88592d8d65bd7468226fad5f8eea84545ed3da: Status 404 returned error can't find the container with id f375b025680e93235f06dd00da88592d8d65bd7468226fad5f8eea84545ed3da Jul 28 00:15:05.260579 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.260539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" event={"ID":"4be04558-0ee7-4188-8288-1d8809469a17","Type":"ContainerStarted","Data":"c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94"} Jul 28 00:15:05.260777 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.260605 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" event={"ID":"4be04558-0ee7-4188-8288-1d8809469a17","Type":"ContainerStarted","Data":"a9c12e5582f190dbdfe8d057b65f0b13ede25be00162c271ef1fdec83ecad385"} Jul 28 00:15:05.260777 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.260742 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:05.262099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.262072 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" event={"ID":"4526dd79-f188-44e5-a245-7238c95d29fa","Type":"ContainerStarted","Data":"d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd"} Jul 28 00:15:05.262241 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.262129 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:05.262241 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.262159 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" event={"ID":"4526dd79-f188-44e5-a245-7238c95d29fa","Type":"ContainerStarted","Data":"f375b025680e93235f06dd00da88592d8d65bd7468226fad5f8eea84545ed3da"} Jul 28 00:15:05.263305 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.263285 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" event={"ID":"010d41f1-9c5d-4da2-8222-fcc4c4848c5f","Type":"ContainerStarted","Data":"bafb53421f40c3de610625de45791f65d1e82e32ee825da010f2467596b4e8fb"} Jul 28 00:15:05.275127 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.275065 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" podStartSLOduration=1.275049688 podStartE2EDuration="1.275049688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:05.273266882 +0000 UTC m=+2305.197475762" watchObservedRunningTime="2026-07-28 00:15:05.275049688 +0000 UTC m=+2305.199258566" Jul 28 00:15:05.288900 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:05.288838 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" podStartSLOduration=1.288820545 podStartE2EDuration="1.288820545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:05.28617994 +0000 UTC m=+2305.210388817" watchObservedRunningTime="2026-07-28 00:15:05.288820545 +0000 UTC m=+2305.213029424" Jul 28 00:15:06.272807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:06.272769 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" event={"ID":"010d41f1-9c5d-4da2-8222-fcc4c4848c5f","Type":"ContainerStarted","Data":"f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab"} Jul 28 00:15:06.273222 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:06.272976 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:06.290984 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:06.290926 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" podStartSLOduration=1.7441619579999998 podStartE2EDuration="2.290909431s" totalImagesPullingTime="546.747473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:06.288667406 +0000 UTC m=+2306.212876284" watchObservedRunningTime="2026-07-28 00:15:06.290909431 +0000 UTC m=+2306.215118308" Jul 28 00:15:08.332854 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:08.332789 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:08.838611 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:08.838569 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:09.167867 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:09.167775 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:09.591970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:09.591919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:11.852857 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:11.852816 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:11.932770 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:11.932729 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:11.937204 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:11.937164 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6"] Jul 28 00:15:11.941279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:11.941257 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:11.943465 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:11.943443 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-da47a6-serving-cert\"" Jul 28 00:15:11.953628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:11.953604 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6"] Jul 28 00:15:12.003383 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.003344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:12.052753 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.052709 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89z9q\" (UniqueName: \"kubernetes.io/projected/30ef919b-9922-4028-b846-9960e969e829-kube-api-access-89z9q\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.052949 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.052795 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ef919b-9922-4028-b846-9960e969e829-openshift-service-ca\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.052949 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.052838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ef919b-9922-4028-b846-9960e969e829-maas-api-tls\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.153625 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.153526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ef919b-9922-4028-b846-9960e969e829-openshift-service-ca\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.153625 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.153589 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ef919b-9922-4028-b846-9960e969e829-maas-api-tls\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.153625 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.153624 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89z9q\" (UniqueName: \"kubernetes.io/projected/30ef919b-9922-4028-b846-9960e969e829-kube-api-access-89z9q\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.154307 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.154279 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ef919b-9922-4028-b846-9960e969e829-openshift-service-ca\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.156166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.156144 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ef919b-9922-4028-b846-9960e969e829-maas-api-tls\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.161453 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.161427 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89z9q\" (UniqueName: \"kubernetes.io/projected/30ef919b-9922-4028-b846-9960e969e829-kube-api-access-89z9q\") pod \"maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.161937 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.161907 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:12.252705 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.252671 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:12.283136 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.283109 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:12.396899 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.396872 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6"] Jul 28 00:15:12.400030 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:12.399999 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ef919b_9922_4028_b846_9960e969e829.slice/crio-6b9d59a2253fca66e41ba53ed92d8613292eb535b6bd3f464bf82a824fa2e641 WatchSource:0}: Error finding container 6b9d59a2253fca66e41ba53ed92d8613292eb535b6bd3f464bf82a824fa2e641: Status 404 returned error can't find the container with id 6b9d59a2253fca66e41ba53ed92d8613292eb535b6bd3f464bf82a824fa2e641 Jul 28 00:15:12.607702 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.607668 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:12.692826 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.692744 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:12.697334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.697304 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp"] Jul 28 00:15:12.702294 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.702271 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.704298 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.704274 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-da47a6\"" Jul 28 00:15:12.704423 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.704322 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-da47a6-dockercfg-86d8q\"" Jul 28 00:15:12.710800 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.710772 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp"] Jul 28 00:15:12.754102 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.754065 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc"] Jul 28 00:15:12.757484 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.757467 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:12.759243 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.759218 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26km2\" (UniqueName: \"kubernetes.io/projected/25e8f49a-a13d-459a-b489-702fe48dbcac-kube-api-access-26km2\") pod \"payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.759369 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.759325 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e8f49a-a13d-459a-b489-702fe48dbcac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.770124 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.770102 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc"] Jul 28 00:15:12.860136 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.860105 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkjr\" (UniqueName: \"kubernetes.io/projected/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-kube-api-access-xmkjr\") pod \"payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:12.860439 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.860145 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e8f49a-a13d-459a-b489-702fe48dbcac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.860439 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.860195 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:12.860439 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.860215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26km2\" (UniqueName: \"kubernetes.io/projected/25e8f49a-a13d-459a-b489-702fe48dbcac-kube-api-access-26km2\") pod \"payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.860829 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.860806 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e8f49a-a13d-459a-b489-702fe48dbcac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.882350 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.882299 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26km2\" (UniqueName: \"kubernetes.io/projected/25e8f49a-a13d-459a-b489-702fe48dbcac-kube-api-access-26km2\") pod \"payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:12.961333 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.961245 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:12.961333 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.961329 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkjr\" (UniqueName: \"kubernetes.io/projected/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-kube-api-access-xmkjr\") pod \"payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:12.961974 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.961952 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:12.968654 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:12.968626 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkjr\" (UniqueName: \"kubernetes.io/projected/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-kube-api-access-xmkjr\") pod \"payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:13.014686 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.014644 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:13.068070 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.068035 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:13.166929 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.166898 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp"] Jul 28 00:15:13.230349 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.230199 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc"] Jul 28 00:15:13.232437 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:13.232406 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171ec359_9b00_4bcd_9ae2_ce8bf8e4d203.slice/crio-72ef271bf72088e27c9464fc1b7c249393e414200e04b5937cb31600b77beb28 WatchSource:0}: Error finding container 72ef271bf72088e27c9464fc1b7c249393e414200e04b5937cb31600b77beb28: Status 404 returned error can't find the container with id 72ef271bf72088e27c9464fc1b7c249393e414200e04b5937cb31600b77beb28 Jul 28 00:15:13.312095 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.312063 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" event={"ID":"30ef919b-9922-4028-b846-9960e969e829","Type":"ContainerStarted","Data":"32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4"} Jul 28 00:15:13.312261 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.312119 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" event={"ID":"30ef919b-9922-4028-b846-9960e969e829","Type":"ContainerStarted","Data":"6b9d59a2253fca66e41ba53ed92d8613292eb535b6bd3f464bf82a824fa2e641"} Jul 28 00:15:13.312261 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.312151 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:13.313824 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.313798 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" event={"ID":"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203","Type":"ContainerStarted","Data":"237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998"} Jul 28 00:15:13.313947 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.313845 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" event={"ID":"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203","Type":"ContainerStarted","Data":"72ef271bf72088e27c9464fc1b7c249393e414200e04b5937cb31600b77beb28"} Jul 28 00:15:13.313947 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.313876 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:13.315370 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.315348 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" event={"ID":"25e8f49a-a13d-459a-b489-702fe48dbcac","Type":"ContainerStarted","Data":"2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751"} Jul 28 00:15:13.315488 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.315384 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" event={"ID":"25e8f49a-a13d-459a-b489-702fe48dbcac","Type":"ContainerStarted","Data":"320da7a416f07815bf40f0a5b404c3086e9f0279f34d806fb35abb20e051c377"} Jul 28 00:15:13.315488 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.315418 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:13.328962 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.328913 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" podStartSLOduration=1.912297261 podStartE2EDuration="2.328897097s" totalImagesPullingTime="416.599836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:13.327397466 +0000 UTC m=+2313.251606346" watchObservedRunningTime="2026-07-28 00:15:13.328897097 +0000 UTC m=+2313.253105974" Jul 28 00:15:13.343786 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.343736 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" podStartSLOduration=1.343720831 podStartE2EDuration="1.343720831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:13.3415796 +0000 UTC m=+2313.265788481" watchObservedRunningTime="2026-07-28 00:15:13.343720831 +0000 UTC m=+2313.267929708" Jul 28 00:15:13.357592 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:13.357548 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" podStartSLOduration=1.357534057 podStartE2EDuration="1.357534057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:13.355406988 +0000 UTC m=+2313.279615867" watchObservedRunningTime="2026-07-28 00:15:13.357534057 +0000 UTC m=+2313.281742935" Jul 28 00:15:16.274671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:16.274642 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:16.275079 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:16.274717 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:16.396697 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:16.396640 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:19.329279 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:19.329248 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:15:24.324530 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.324478 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:15:24.324962 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.324619 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:15:24.730802 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.730719 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72"] Jul 28 00:15:24.740012 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.739979 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:24.740511 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.740467 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72"] Jul 28 00:15:24.742109 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.742086 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:15:24.872808 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.872773 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf1e9e9-a320-4a79-8663-3f171b8ff069-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72\" (UID: \"7bf1e9e9-a320-4a79-8663-3f171b8ff069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:24.973429 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.973387 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf1e9e9-a320-4a79-8663-3f171b8ff069-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72\" (UID: \"7bf1e9e9-a320-4a79-8663-3f171b8ff069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:24.974050 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:24.974027 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf1e9e9-a320-4a79-8663-3f171b8ff069-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72\" (UID: \"7bf1e9e9-a320-4a79-8663-3f171b8ff069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:25.051583 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:25.051545 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:25.190541 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:25.190315 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72"] Jul 28 00:15:25.192981 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:25.192951 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf1e9e9_a320_4a79_8663_3f171b8ff069.slice/crio-430102c4765f29bf6f581336cd53bd3b360b3af491f1422673de84ed4c4e54ff WatchSource:0}: Error finding container 430102c4765f29bf6f581336cd53bd3b360b3af491f1422673de84ed4c4e54ff: Status 404 returned error can't find the container with id 430102c4765f29bf6f581336cd53bd3b360b3af491f1422673de84ed4c4e54ff Jul 28 00:15:25.388220 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:25.388139 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" event={"ID":"7bf1e9e9-a320-4a79-8663-3f171b8ff069","Type":"ContainerStarted","Data":"430102c4765f29bf6f581336cd53bd3b360b3af491f1422673de84ed4c4e54ff"} Jul 28 00:15:26.394376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:26.394327 2555 generic.go:356] "Generic (PLEG): container finished" podID="7bf1e9e9-a320-4a79-8663-3f171b8ff069" containerID="34eb5d921ecf9d3d070fa52485f0d308142917707bf963f35a9f671ac4cd31ce" exitCode=0 Jul 28 00:15:26.394814 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:26.394374 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" event={"ID":"7bf1e9e9-a320-4a79-8663-3f171b8ff069","Type":"ContainerDied","Data":"34eb5d921ecf9d3d070fa52485f0d308142917707bf963f35a9f671ac4cd31ce"} Jul 28 00:15:27.543888 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:27.543860 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:27.697368 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:27.697267 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf1e9e9-a320-4a79-8663-3f171b8ff069-maas-api-service-ca\") pod \"7bf1e9e9-a320-4a79-8663-3f171b8ff069\" (UID: \"7bf1e9e9-a320-4a79-8663-3f171b8ff069\") " Jul 28 00:15:27.697713 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:27.697687 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf1e9e9-a320-4a79-8663-3f171b8ff069-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bf1e9e9-a320-4a79-8663-3f171b8ff069" (UID: "7bf1e9e9-a320-4a79-8663-3f171b8ff069"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:15:27.798693 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:27.798657 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf1e9e9-a320-4a79-8663-3f171b8ff069-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:28.406078 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:28.406044 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" Jul 28 00:15:28.406264 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:28.406080 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72" event={"ID":"7bf1e9e9-a320-4a79-8663-3f171b8ff069","Type":"ContainerDied","Data":"430102c4765f29bf6f581336cd53bd3b360b3af491f1422673de84ed4c4e54ff"} Jul 28 00:15:28.406264 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:28.406126 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="430102c4765f29bf6f581336cd53bd3b360b3af491f1422673de84ed4c4e54ff" Jul 28 00:15:40.784508 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.784415 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7"] Jul 28 00:15:40.784966 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.784798 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" podUID="010d41f1-9c5d-4da2-8222-fcc4c4848c5f" containerName="maas-api" containerID="cri-o://f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab" gracePeriod=30 Jul 28 00:15:40.806041 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.806007 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:40.833550 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.833512 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq"] Jul 28 00:15:40.833805 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.833777 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" podUID="4526dd79-f188-44e5-a245-7238c95d29fa" containerName="payload-processing" containerID="cri-o://d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd" gracePeriod=30 Jul 28 00:15:40.838929 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.838900 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs"] Jul 28 00:15:40.839301 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.839271 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" podUID="4be04558-0ee7-4188-8288-1d8809469a17" containerName="payload-pre-processing" containerID="cri-o://c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94" gracePeriod=30 Jul 28 00:15:40.871351 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.871318 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:40.930725 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:40.930692 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:41.031792 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.031751 2555 scope.go:117] "RemoveContainer" containerID="a08508c17c682ecaa6d7735f2517a6788da4a09559051d1c3f3e27ffc2a39907" Jul 28 00:15:41.145781 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.145761 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:41.147827 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.147803 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:41.150917 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.150895 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:41.216842 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.216815 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-openshift-service-ca\") pod \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " Jul 28 00:15:41.217016 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.216892 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sxgl\" (UniqueName: \"kubernetes.io/projected/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-kube-api-access-8sxgl\") pod \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " Jul 28 00:15:41.217016 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.216927 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7llwn\" (UniqueName: \"kubernetes.io/projected/4526dd79-f188-44e5-a245-7238c95d29fa-kube-api-access-7llwn\") pod \"4526dd79-f188-44e5-a245-7238c95d29fa\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " Jul 28 00:15:41.217016 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.216942 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be04558-0ee7-4188-8288-1d8809469a17-plugins-config-volume\") pod \"4be04558-0ee7-4188-8288-1d8809469a17\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " Jul 28 00:15:41.217016 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.216976 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk8pv\" (UniqueName: \"kubernetes.io/projected/4be04558-0ee7-4188-8288-1d8809469a17-kube-api-access-fk8pv\") pod \"4be04558-0ee7-4188-8288-1d8809469a17\" (UID: \"4be04558-0ee7-4188-8288-1d8809469a17\") " Jul 28 00:15:41.217254 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.217017 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls\") pod \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\" (UID: \"010d41f1-9c5d-4da2-8222-fcc4c4848c5f\") " Jul 28 00:15:41.217254 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.217067 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526dd79-f188-44e5-a245-7238c95d29fa-plugins-config-volume\") pod \"4526dd79-f188-44e5-a245-7238c95d29fa\" (UID: \"4526dd79-f188-44e5-a245-7238c95d29fa\") " Jul 28 00:15:41.217254 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.217160 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "010d41f1-9c5d-4da2-8222-fcc4c4848c5f" (UID: "010d41f1-9c5d-4da2-8222-fcc4c4848c5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:15:41.217418 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.217355 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be04558-0ee7-4188-8288-1d8809469a17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4be04558-0ee7-4188-8288-1d8809469a17" (UID: "4be04558-0ee7-4188-8288-1d8809469a17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:15:41.217418 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.217371 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.217632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.217602 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526dd79-f188-44e5-a245-7238c95d29fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4526dd79-f188-44e5-a245-7238c95d29fa" (UID: "4526dd79-f188-44e5-a245-7238c95d29fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:15:41.219124 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.219103 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4526dd79-f188-44e5-a245-7238c95d29fa-kube-api-access-7llwn" (OuterVolumeSpecName: "kube-api-access-7llwn") pod "4526dd79-f188-44e5-a245-7238c95d29fa" (UID: "4526dd79-f188-44e5-a245-7238c95d29fa"). InnerVolumeSpecName "kube-api-access-7llwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:15:41.219215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.219171 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "010d41f1-9c5d-4da2-8222-fcc4c4848c5f" (UID: "010d41f1-9c5d-4da2-8222-fcc4c4848c5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:15:41.219464 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.219447 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-kube-api-access-8sxgl" (OuterVolumeSpecName: "kube-api-access-8sxgl") pod "010d41f1-9c5d-4da2-8222-fcc4c4848c5f" (UID: "010d41f1-9c5d-4da2-8222-fcc4c4848c5f"). InnerVolumeSpecName "kube-api-access-8sxgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:15:41.219542 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.219460 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be04558-0ee7-4188-8288-1d8809469a17-kube-api-access-fk8pv" (OuterVolumeSpecName: "kube-api-access-fk8pv") pod "4be04558-0ee7-4188-8288-1d8809469a17" (UID: "4be04558-0ee7-4188-8288-1d8809469a17"). InnerVolumeSpecName "kube-api-access-fk8pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:15:41.318320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.318241 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llwn\" (UniqueName: \"kubernetes.io/projected/4526dd79-f188-44e5-a245-7238c95d29fa-kube-api-access-7llwn\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.318320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.318270 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be04558-0ee7-4188-8288-1d8809469a17-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.318320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.318280 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8pv\" (UniqueName: \"kubernetes.io/projected/4be04558-0ee7-4188-8288-1d8809469a17-kube-api-access-fk8pv\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.318320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.318289 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.318320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.318298 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526dd79-f188-44e5-a245-7238c95d29fa-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.318320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.318307 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sxgl\" (UniqueName: \"kubernetes.io/projected/010d41f1-9c5d-4da2-8222-fcc4c4848c5f-kube-api-access-8sxgl\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:41.480798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.480762 2555 generic.go:356] "Generic (PLEG): container finished" podID="010d41f1-9c5d-4da2-8222-fcc4c4848c5f" containerID="f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab" exitCode=0 Jul 28 00:15:41.480991 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.480826 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" Jul 28 00:15:41.480991 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.480858 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" event={"ID":"010d41f1-9c5d-4da2-8222-fcc4c4848c5f","Type":"ContainerDied","Data":"f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab"} Jul 28 00:15:41.480991 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.480920 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7" event={"ID":"010d41f1-9c5d-4da2-8222-fcc4c4848c5f","Type":"ContainerDied","Data":"bafb53421f40c3de610625de45791f65d1e82e32ee825da010f2467596b4e8fb"} Jul 28 00:15:41.480991 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.480951 2555 scope.go:117] "RemoveContainer" containerID="f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab" Jul 28 00:15:41.482411 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.482253 2555 generic.go:356] "Generic (PLEG): container finished" podID="4526dd79-f188-44e5-a245-7238c95d29fa" containerID="d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd" exitCode=0 Jul 28 00:15:41.482411 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.482302 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" Jul 28 00:15:41.482411 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.482328 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" event={"ID":"4526dd79-f188-44e5-a245-7238c95d29fa","Type":"ContainerDied","Data":"d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd"} Jul 28 00:15:41.482411 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.482389 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq" event={"ID":"4526dd79-f188-44e5-a245-7238c95d29fa","Type":"ContainerDied","Data":"f375b025680e93235f06dd00da88592d8d65bd7468226fad5f8eea84545ed3da"} Jul 28 00:15:41.483962 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.483938 2555 generic.go:356] "Generic (PLEG): container finished" podID="4be04558-0ee7-4188-8288-1d8809469a17" containerID="c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94" exitCode=0 Jul 28 00:15:41.484099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.483984 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" Jul 28 00:15:41.484099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.484016 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" event={"ID":"4be04558-0ee7-4188-8288-1d8809469a17","Type":"ContainerDied","Data":"c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94"} Jul 28 00:15:41.484099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.484077 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs" event={"ID":"4be04558-0ee7-4188-8288-1d8809469a17","Type":"ContainerDied","Data":"a9c12e5582f190dbdfe8d057b65f0b13ede25be00162c271ef1fdec83ecad385"} Jul 28 00:15:41.491379 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.491360 2555 scope.go:117] "RemoveContainer" containerID="f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab" Jul 28 00:15:41.491777 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:15:41.491752 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab\": container with ID starting with f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab not found: ID does not exist" containerID="f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab" Jul 28 00:15:41.491888 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.491787 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab"} err="failed to get container status \"f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab\": rpc error: code = NotFound desc = could not find container \"f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab\": container with ID starting with f47560421c1d84539f4526baeb825cafc8093d44c0afc3aa6f99b48951b69cab not found: ID does not exist" Jul 28 00:15:41.491888 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.491804 2555 scope.go:117] "RemoveContainer" containerID="d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd" Jul 28 00:15:41.499536 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.499518 2555 scope.go:117] "RemoveContainer" containerID="d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd" Jul 28 00:15:41.499792 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:15:41.499772 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd\": container with ID starting with d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd not found: ID does not exist" containerID="d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd" Jul 28 00:15:41.499878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.499797 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd"} err="failed to get container status \"d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd\": rpc error: code = NotFound desc = could not find container \"d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd\": container with ID starting with d74e54bb4267a9a564c943cecdbe8374fafd199f2fa4f6e733fc8484cdb950bd not found: ID does not exist" Jul 28 00:15:41.499878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.499812 2555 scope.go:117] "RemoveContainer" containerID="c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94" Jul 28 00:15:41.506312 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.506292 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7"] Jul 28 00:15:41.507947 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.507690 2555 scope.go:117] "RemoveContainer" containerID="c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94" Jul 28 00:15:41.508101 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:15:41.508081 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94\": container with ID starting with c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94 not found: ID does not exist" containerID="c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94" Jul 28 00:15:41.508173 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.508111 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94"} err="failed to get container status \"c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94\": rpc error: code = NotFound desc = could not find container \"c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94\": container with ID starting with c838cf86a1bc1c416f42868ca9c6ff88193b0b4b94be0276da76488562e7fe94 not found: ID does not exist" Jul 28 00:15:41.509771 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.509751 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd61a5-5675595558-w6dr7"] Jul 28 00:15:41.518717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.518698 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs"] Jul 28 00:15:41.520642 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.520624 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd61a5-799ffdcd88-llbrs"] Jul 28 00:15:41.529720 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.529697 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq"] Jul 28 00:15:41.531483 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:41.531461 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd61a5-5988b5dd84-s8tgq"] Jul 28 00:15:42.564742 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:42.564708 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="010d41f1-9c5d-4da2-8222-fcc4c4848c5f" path="/var/lib/kubelet/pods/010d41f1-9c5d-4da2-8222-fcc4c4848c5f/volumes" Jul 28 00:15:42.565125 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:42.565073 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4526dd79-f188-44e5-a245-7238c95d29fa" path="/var/lib/kubelet/pods/4526dd79-f188-44e5-a245-7238c95d29fa/volumes" Jul 28 00:15:42.565381 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:42.565368 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be04558-0ee7-4188-8288-1d8809469a17" path="/var/lib/kubelet/pods/4be04558-0ee7-4188-8288-1d8809469a17/volumes" Jul 28 00:15:44.798427 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:44.798391 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72"] Jul 28 00:15:44.801699 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:44.801663 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:44.801848 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:44.801679 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd61a5-p4g72"] Jul 28 00:15:45.226527 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:45.226400 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:46.565043 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:46.565011 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf1e9e9-a320-4a79-8663-3f171b8ff069" path="/var/lib/kubelet/pods/7bf1e9e9-a320-4a79-8663-3f171b8ff069/volumes" Jul 28 00:15:50.983646 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:50.983603 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:51.087406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.087363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:51.399355 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.399318 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:51.527018 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.526981 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:15:51.655739 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.655655 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp"] Jul 28 00:15:51.656238 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656222 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4526dd79-f188-44e5-a245-7238c95d29fa" containerName="payload-processing" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656240 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4526dd79-f188-44e5-a245-7238c95d29fa" containerName="payload-processing" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656265 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be04558-0ee7-4188-8288-1d8809469a17" containerName="payload-pre-processing" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656275 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be04558-0ee7-4188-8288-1d8809469a17" containerName="payload-pre-processing" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656294 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010d41f1-9c5d-4da2-8222-fcc4c4848c5f" containerName="maas-api" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656303 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="010d41f1-9c5d-4da2-8222-fcc4c4848c5f" containerName="maas-api" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656312 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf1e9e9-a320-4a79-8663-3f171b8ff069" containerName="revoke-keys" Jul 28 00:15:51.656320 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656321 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf1e9e9-a320-4a79-8663-3f171b8ff069" containerName="revoke-keys" Jul 28 00:15:51.656716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656413 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="010d41f1-9c5d-4da2-8222-fcc4c4848c5f" containerName="maas-api" Jul 28 00:15:51.656716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656426 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4526dd79-f188-44e5-a245-7238c95d29fa" containerName="payload-processing" Jul 28 00:15:51.656716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656435 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be04558-0ee7-4188-8288-1d8809469a17" containerName="payload-pre-processing" Jul 28 00:15:51.656716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.656447 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf1e9e9-a320-4a79-8663-3f171b8ff069" containerName="revoke-keys" Jul 28 00:15:51.660966 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.660943 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:51.663368 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.663348 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:15:51.665175 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.665153 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp"] Jul 28 00:15:51.702904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.702867 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a9bcb-482c-48b7-a424-f21ba56b28c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp\" (UID: \"a47a9bcb-482c-48b7-a424-f21ba56b28c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:51.804109 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.804082 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a9bcb-482c-48b7-a424-f21ba56b28c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp\" (UID: \"a47a9bcb-482c-48b7-a424-f21ba56b28c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:51.804719 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.804698 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a9bcb-482c-48b7-a424-f21ba56b28c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp\" (UID: \"a47a9bcb-482c-48b7-a424-f21ba56b28c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:51.976034 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:51.975959 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:52.113945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:52.113919 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp"] Jul 28 00:15:52.117411 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:15:52.117379 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47a9bcb_482c_48b7_a424_f21ba56b28c3.slice/crio-7b62943bc18b11122822faaa71f49f9dc1ede18d4ce1526e0b65f82c7fe9c85a WatchSource:0}: Error finding container 7b62943bc18b11122822faaa71f49f9dc1ede18d4ce1526e0b65f82c7fe9c85a: Status 404 returned error can't find the container with id 7b62943bc18b11122822faaa71f49f9dc1ede18d4ce1526e0b65f82c7fe9c85a Jul 28 00:15:52.546701 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:52.546614 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" event={"ID":"a47a9bcb-482c-48b7-a424-f21ba56b28c3","Type":"ContainerStarted","Data":"7fb76e825404157c9ee91d69e9d5ff7c375e71cf4750a116ba889e90b8c35b6e"} Jul 28 00:15:52.546701 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:52.546665 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" event={"ID":"a47a9bcb-482c-48b7-a424-f21ba56b28c3","Type":"ContainerStarted","Data":"7b62943bc18b11122822faaa71f49f9dc1ede18d4ce1526e0b65f82c7fe9c85a"} Jul 28 00:15:52.561832 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:52.561781 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" podStartSLOduration=1.561766886 podStartE2EDuration="1.561766886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:15:52.561151741 +0000 UTC m=+2352.485360620" watchObservedRunningTime="2026-07-28 00:15:52.561766886 +0000 UTC m=+2352.485975764" Jul 28 00:15:53.552481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:53.552444 2555 generic.go:356] "Generic (PLEG): container finished" podID="a47a9bcb-482c-48b7-a424-f21ba56b28c3" containerID="7fb76e825404157c9ee91d69e9d5ff7c375e71cf4750a116ba889e90b8c35b6e" exitCode=0 Jul 28 00:15:53.552883 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:53.552535 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" event={"ID":"a47a9bcb-482c-48b7-a424-f21ba56b28c3","Type":"ContainerDied","Data":"7fb76e825404157c9ee91d69e9d5ff7c375e71cf4750a116ba889e90b8c35b6e"} Jul 28 00:15:54.689510 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:54.689475 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:54.728607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:54.728574 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a9bcb-482c-48b7-a424-f21ba56b28c3-maas-api-service-ca\") pod \"a47a9bcb-482c-48b7-a424-f21ba56b28c3\" (UID: \"a47a9bcb-482c-48b7-a424-f21ba56b28c3\") " Jul 28 00:15:54.728930 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:54.728911 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47a9bcb-482c-48b7-a424-f21ba56b28c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a47a9bcb-482c-48b7-a424-f21ba56b28c3" (UID: "a47a9bcb-482c-48b7-a424-f21ba56b28c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:15:54.829985 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:54.829906 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a9bcb-482c-48b7-a424-f21ba56b28c3-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:15:55.563092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:55.563064 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" Jul 28 00:15:55.563092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:55.563086 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp" event={"ID":"a47a9bcb-482c-48b7-a424-f21ba56b28c3","Type":"ContainerDied","Data":"7b62943bc18b11122822faaa71f49f9dc1ede18d4ce1526e0b65f82c7fe9c85a"} Jul 28 00:15:55.563310 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:15:55.563133 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b62943bc18b11122822faaa71f49f9dc1ede18d4ce1526e0b65f82c7fe9c85a" Jul 28 00:16:07.045346 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.045312 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6"] Jul 28 00:16:07.045834 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.045554 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" podUID="30ef919b-9922-4028-b846-9960e969e829" containerName="maas-api" containerID="cri-o://32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4" gracePeriod=30 Jul 28 00:16:07.080397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.080361 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:07.096335 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.096310 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc"] Jul 28 00:16:07.096545 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.096523 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" podUID="171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" containerName="payload-processing" containerID="cri-o://237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998" gracePeriod=30 Jul 28 00:16:07.104115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.104090 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp"] Jul 28 00:16:07.104313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.104291 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" podUID="25e8f49a-a13d-459a-b489-702fe48dbcac" containerName="payload-pre-processing" containerID="cri-o://2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751" gracePeriod=30 Jul 28 00:16:07.178670 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.178634 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:07.330796 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.330767 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:07.370127 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.370104 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:16:07.373660 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.373623 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:16:07.413174 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.413142 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:16:07.439729 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.439693 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89z9q\" (UniqueName: \"kubernetes.io/projected/30ef919b-9922-4028-b846-9960e969e829-kube-api-access-89z9q\") pod \"30ef919b-9922-4028-b846-9960e969e829\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " Jul 28 00:16:07.439899 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.439764 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-plugins-config-volume\") pod \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " Jul 28 00:16:07.439899 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.439803 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmkjr\" (UniqueName: \"kubernetes.io/projected/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-kube-api-access-xmkjr\") pod \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\" (UID: \"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203\") " Jul 28 00:16:07.439899 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.439887 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ef919b-9922-4028-b846-9960e969e829-maas-api-tls\") pod \"30ef919b-9922-4028-b846-9960e969e829\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " Jul 28 00:16:07.440064 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.439926 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ef919b-9922-4028-b846-9960e969e829-openshift-service-ca\") pod \"30ef919b-9922-4028-b846-9960e969e829\" (UID: \"30ef919b-9922-4028-b846-9960e969e829\") " Jul 28 00:16:07.440291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.440158 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" (UID: "171ec359-9b00-4bcd-9ae2-ce8bf8e4d203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:16:07.440436 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.440318 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ef919b-9922-4028-b846-9960e969e829-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30ef919b-9922-4028-b846-9960e969e829" (UID: "30ef919b-9922-4028-b846-9960e969e829"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:16:07.442256 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.442229 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ef919b-9922-4028-b846-9960e969e829-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30ef919b-9922-4028-b846-9960e969e829" (UID: "30ef919b-9922-4028-b846-9960e969e829"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:16:07.442374 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.442353 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-kube-api-access-xmkjr" (OuterVolumeSpecName: "kube-api-access-xmkjr") pod "171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" (UID: "171ec359-9b00-4bcd-9ae2-ce8bf8e4d203"). InnerVolumeSpecName "kube-api-access-xmkjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:16:07.442462 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.442366 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ef919b-9922-4028-b846-9960e969e829-kube-api-access-89z9q" (OuterVolumeSpecName: "kube-api-access-89z9q") pod "30ef919b-9922-4028-b846-9960e969e829" (UID: "30ef919b-9922-4028-b846-9960e969e829"). InnerVolumeSpecName "kube-api-access-89z9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:16:07.541220 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541182 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26km2\" (UniqueName: \"kubernetes.io/projected/25e8f49a-a13d-459a-b489-702fe48dbcac-kube-api-access-26km2\") pod \"25e8f49a-a13d-459a-b489-702fe48dbcac\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " Jul 28 00:16:07.541401 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541276 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e8f49a-a13d-459a-b489-702fe48dbcac-plugins-config-volume\") pod \"25e8f49a-a13d-459a-b489-702fe48dbcac\" (UID: \"25e8f49a-a13d-459a-b489-702fe48dbcac\") " Jul 28 00:16:07.541527 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541486 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ef919b-9922-4028-b846-9960e969e829-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.541569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541533 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ef919b-9922-4028-b846-9960e969e829-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.541569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541550 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89z9q\" (UniqueName: \"kubernetes.io/projected/30ef919b-9922-4028-b846-9960e969e829-kube-api-access-89z9q\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.541569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541564 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.541666 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541578 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmkjr\" (UniqueName: \"kubernetes.io/projected/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203-kube-api-access-xmkjr\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.541700 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.541672 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e8f49a-a13d-459a-b489-702fe48dbcac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25e8f49a-a13d-459a-b489-702fe48dbcac" (UID: "25e8f49a-a13d-459a-b489-702fe48dbcac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:16:07.543220 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.543198 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e8f49a-a13d-459a-b489-702fe48dbcac-kube-api-access-26km2" (OuterVolumeSpecName: "kube-api-access-26km2") pod "25e8f49a-a13d-459a-b489-702fe48dbcac" (UID: "25e8f49a-a13d-459a-b489-702fe48dbcac"). InnerVolumeSpecName "kube-api-access-26km2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:16:07.622037 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.621937 2555 generic.go:356] "Generic (PLEG): container finished" podID="25e8f49a-a13d-459a-b489-702fe48dbcac" containerID="2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751" exitCode=0 Jul 28 00:16:07.622037 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.621981 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" Jul 28 00:16:07.622244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.622033 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" event={"ID":"25e8f49a-a13d-459a-b489-702fe48dbcac","Type":"ContainerDied","Data":"2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751"} Jul 28 00:16:07.622244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.622089 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp" event={"ID":"25e8f49a-a13d-459a-b489-702fe48dbcac","Type":"ContainerDied","Data":"320da7a416f07815bf40f0a5b404c3086e9f0279f34d806fb35abb20e051c377"} Jul 28 00:16:07.622244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.622117 2555 scope.go:117] "RemoveContainer" containerID="2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751" Jul 28 00:16:07.623419 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.623387 2555 generic.go:356] "Generic (PLEG): container finished" podID="30ef919b-9922-4028-b846-9960e969e829" containerID="32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4" exitCode=0 Jul 28 00:16:07.623553 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.623437 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" Jul 28 00:16:07.623553 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.623479 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" event={"ID":"30ef919b-9922-4028-b846-9960e969e829","Type":"ContainerDied","Data":"32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4"} Jul 28 00:16:07.623553 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.623546 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6" event={"ID":"30ef919b-9922-4028-b846-9960e969e829","Type":"ContainerDied","Data":"6b9d59a2253fca66e41ba53ed92d8613292eb535b6bd3f464bf82a824fa2e641"} Jul 28 00:16:07.624944 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.624921 2555 generic.go:356] "Generic (PLEG): container finished" podID="171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" containerID="237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998" exitCode=0 Jul 28 00:16:07.625077 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.624957 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" event={"ID":"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203","Type":"ContainerDied","Data":"237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998"} Jul 28 00:16:07.625077 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.624987 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" event={"ID":"171ec359-9b00-4bcd-9ae2-ce8bf8e4d203","Type":"ContainerDied","Data":"72ef271bf72088e27c9464fc1b7c249393e414200e04b5937cb31600b77beb28"} Jul 28 00:16:07.625077 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.624999 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc" Jul 28 00:16:07.631970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.631953 2555 scope.go:117] "RemoveContainer" containerID="2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751" Jul 28 00:16:07.632311 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:16:07.632287 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751\": container with ID starting with 2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751 not found: ID does not exist" containerID="2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751" Jul 28 00:16:07.632413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.632323 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751"} err="failed to get container status \"2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751\": rpc error: code = NotFound desc = could not find container \"2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751\": container with ID starting with 2ea7656788b0caa953c1834b05a79d2588430212c3a2b95f747b485c01f98751 not found: ID does not exist" Jul 28 00:16:07.632413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.632347 2555 scope.go:117] "RemoveContainer" containerID="32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4" Jul 28 00:16:07.641399 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.641384 2555 scope.go:117] "RemoveContainer" containerID="32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4" Jul 28 00:16:07.641650 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:16:07.641626 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4\": container with ID starting with 32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4 not found: ID does not exist" containerID="32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4" Jul 28 00:16:07.641739 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.641658 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4"} err="failed to get container status \"32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4\": rpc error: code = NotFound desc = could not find container \"32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4\": container with ID starting with 32d0a7385f66f4b71c681876762fdb5ada987af28c0367773ba97492452e12c4 not found: ID does not exist" Jul 28 00:16:07.641739 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.641680 2555 scope.go:117] "RemoveContainer" containerID="237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998" Jul 28 00:16:07.641942 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.641924 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e8f49a-a13d-459a-b489-702fe48dbcac-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.641989 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.641947 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26km2\" (UniqueName: \"kubernetes.io/projected/25e8f49a-a13d-459a-b489-702fe48dbcac-kube-api-access-26km2\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:16:07.647880 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.647858 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp"] Jul 28 00:16:07.649359 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.649326 2555 scope.go:117] "RemoveContainer" containerID="237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998" Jul 28 00:16:07.649776 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:16:07.649756 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998\": container with ID starting with 237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998 not found: ID does not exist" containerID="237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998" Jul 28 00:16:07.649854 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.649781 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998"} err="failed to get container status \"237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998\": rpc error: code = NotFound desc = could not find container \"237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998\": container with ID starting with 237909ac5c7c0ab3ee7eef8a6b2a516fc57a9a7ff0d3626edc7e6397939e8998 not found: ID does not exist" Jul 28 00:16:07.651360 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.651342 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da47a6-6d498db9cb-pwdrp"] Jul 28 00:16:07.661224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.661204 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6"] Jul 28 00:16:07.662898 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.662880 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da47a6-567fd96c7b-phfb6"] Jul 28 00:16:07.671606 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.671584 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc"] Jul 28 00:16:07.675018 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:07.674998 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da47a6-77ff7f84b4-wghlc"] Jul 28 00:16:08.565032 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:08.565002 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" path="/var/lib/kubelet/pods/171ec359-9b00-4bcd-9ae2-ce8bf8e4d203/volumes" Jul 28 00:16:08.565445 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:08.565337 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e8f49a-a13d-459a-b489-702fe48dbcac" path="/var/lib/kubelet/pods/25e8f49a-a13d-459a-b489-702fe48dbcac/volumes" Jul 28 00:16:08.565690 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:08.565677 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ef919b-9922-4028-b846-9960e969e829" path="/var/lib/kubelet/pods/30ef919b-9922-4028-b846-9960e969e829/volumes" Jul 28 00:16:11.733573 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:11.733529 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:11.733990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:11.733587 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp"] Jul 28 00:16:11.736124 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:11.736101 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da47a6-mhrsp"] Jul 28 00:16:12.209278 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:12.209247 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:12.565677 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:12.565643 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47a9bcb-482c-48b7-a424-f21ba56b28c3" path="/var/lib/kubelet/pods/a47a9bcb-482c-48b7-a424-f21ba56b28c3/volumes" Jul 28 00:16:17.872864 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:17.872821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:17.973207 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:17.973169 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:18.153530 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:18.153423 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:18.390071 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:18.390035 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:18.742525 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:18.742480 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:19.074181 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:19.074096 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:19.230982 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:19.230938 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:22.225133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.225091 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:22.329579 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.329543 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:22.335340 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335312 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7"] Jul 28 00:16:22.335701 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335686 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ef919b-9922-4028-b846-9960e969e829" containerName="maas-api" Jul 28 00:16:22.335701 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335702 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ef919b-9922-4028-b846-9960e969e829" containerName="maas-api" Jul 28 00:16:22.335795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335710 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47a9bcb-482c-48b7-a424-f21ba56b28c3" containerName="revoke-keys" Jul 28 00:16:22.335795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335716 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47a9bcb-482c-48b7-a424-f21ba56b28c3" containerName="revoke-keys" Jul 28 00:16:22.335795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335729 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e8f49a-a13d-459a-b489-702fe48dbcac" containerName="payload-pre-processing" Jul 28 00:16:22.335795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335735 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e8f49a-a13d-459a-b489-702fe48dbcac" containerName="payload-pre-processing" Jul 28 00:16:22.335795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335746 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" containerName="payload-processing" Jul 28 00:16:22.335795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335751 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" containerName="payload-processing" Jul 28 00:16:22.335978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335822 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ef919b-9922-4028-b846-9960e969e829" containerName="maas-api" Jul 28 00:16:22.335978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335830 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47a9bcb-482c-48b7-a424-f21ba56b28c3" containerName="revoke-keys" Jul 28 00:16:22.335978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335836 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="171ec359-9b00-4bcd-9ae2-ce8bf8e4d203" containerName="payload-processing" Jul 28 00:16:22.335978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.335843 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e8f49a-a13d-459a-b489-702fe48dbcac" containerName="payload-pre-processing" Jul 28 00:16:22.338800 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.338781 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.340900 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.340880 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-328ef1-serving-cert\"" Jul 28 00:16:22.353904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.353880 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7"] Jul 28 00:16:22.446834 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.446792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:22.470145 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.470112 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj7z\" (UniqueName: \"kubernetes.io/projected/e9f224be-5549-477b-ac38-522232c4c761-kube-api-access-klj7z\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.470297 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.470162 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f224be-5549-477b-ac38-522232c4c761-maas-api-tls\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.470297 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.470234 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f224be-5549-477b-ac38-522232c4c761-openshift-service-ca\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.542910 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.542873 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:22.570967 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.570944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f224be-5549-477b-ac38-522232c4c761-openshift-service-ca\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.571115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.570998 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj7z\" (UniqueName: \"kubernetes.io/projected/e9f224be-5549-477b-ac38-522232c4c761-kube-api-access-klj7z\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.571115 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.571033 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f224be-5549-477b-ac38-522232c4c761-maas-api-tls\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.571571 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.571553 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f224be-5549-477b-ac38-522232c4c761-openshift-service-ca\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.573361 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.573343 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f224be-5549-477b-ac38-522232c4c761-maas-api-tls\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.578570 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.578546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj7z\" (UniqueName: \"kubernetes.io/projected/e9f224be-5549-477b-ac38-522232c4c761-kube-api-access-klj7z\") pod \"maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.648737 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.648701 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:22.782482 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.782453 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7"] Jul 28 00:16:22.784822 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:22.784793 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f224be_5549_477b_ac38_522232c4c761.slice/crio-73bf5bdb4bb3d500bfedb7a4ac2d84367c8354501c0295b7dc624ddd461cebea WatchSource:0}: Error finding container 73bf5bdb4bb3d500bfedb7a4ac2d84367c8354501c0295b7dc624ddd461cebea: Status 404 returned error can't find the container with id 73bf5bdb4bb3d500bfedb7a4ac2d84367c8354501c0295b7dc624ddd461cebea Jul 28 00:16:22.941782 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.941702 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:22.985598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.985568 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg"] Jul 28 00:16:22.990062 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.990043 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:22.992325 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.992306 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-328ef1\"" Jul 28 00:16:22.998897 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:22.998877 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg"] Jul 28 00:16:23.011006 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.010979 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:23.030335 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.030315 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg"] Jul 28 00:16:23.033335 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.033319 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.035388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.035362 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-328ef1-dockercfg-hbbfj\"" Jul 28 00:16:23.043133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.043111 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg"] Jul 28 00:16:23.175945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.175915 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjsl\" (UniqueName: \"kubernetes.io/projected/858d61d2-540f-49d6-864a-664fc1a72708-kube-api-access-lqjsl\") pod \"payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.176123 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.175957 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335792df-10dd-4882-81f1-6112414e6631-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.176123 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.175994 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdb7x\" (UniqueName: \"kubernetes.io/projected/335792df-10dd-4882-81f1-6112414e6631-kube-api-access-bdb7x\") pod \"payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.176123 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.176014 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858d61d2-540f-49d6-864a-664fc1a72708-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.277345 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.277312 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335792df-10dd-4882-81f1-6112414e6631-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.277345 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.277346 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdb7x\" (UniqueName: \"kubernetes.io/projected/335792df-10dd-4882-81f1-6112414e6631-kube-api-access-bdb7x\") pod \"payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.277851 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.277365 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858d61d2-540f-49d6-864a-664fc1a72708-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.277851 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.277444 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjsl\" (UniqueName: \"kubernetes.io/projected/858d61d2-540f-49d6-864a-664fc1a72708-kube-api-access-lqjsl\") pod \"payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.278047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.278023 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335792df-10dd-4882-81f1-6112414e6631-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.278169 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.278145 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858d61d2-540f-49d6-864a-664fc1a72708-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.295507 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.295475 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdb7x\" (UniqueName: \"kubernetes.io/projected/335792df-10dd-4882-81f1-6112414e6631-kube-api-access-bdb7x\") pod \"payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.295632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.295525 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjsl\" (UniqueName: \"kubernetes.io/projected/858d61d2-540f-49d6-864a-664fc1a72708-kube-api-access-lqjsl\") pod \"payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.302348 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.302328 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.343613 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.343586 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.457146 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.457115 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg"] Jul 28 00:16:23.459983 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:23.459938 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858d61d2_540f_49d6_864a_664fc1a72708.slice/crio-9d875a03700d651b4aa2254c2ef069dc31359a83893a92cfb5effd24326487f5 WatchSource:0}: Error finding container 9d875a03700d651b4aa2254c2ef069dc31359a83893a92cfb5effd24326487f5: Status 404 returned error can't find the container with id 9d875a03700d651b4aa2254c2ef069dc31359a83893a92cfb5effd24326487f5 Jul 28 00:16:23.502236 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.502211 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg"] Jul 28 00:16:23.503808 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:23.503784 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335792df_10dd_4882_81f1_6112414e6631.slice/crio-13e05cb8aa2f2c2c0b2f753e03e998795328701ef7defcb2b6399182ae1a48bc WatchSource:0}: Error finding container 13e05cb8aa2f2c2c0b2f753e03e998795328701ef7defcb2b6399182ae1a48bc: Status 404 returned error can't find the container with id 13e05cb8aa2f2c2c0b2f753e03e998795328701ef7defcb2b6399182ae1a48bc Jul 28 00:16:23.710289 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.710243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" event={"ID":"335792df-10dd-4882-81f1-6112414e6631","Type":"ContainerStarted","Data":"cff24721d92707a7ac8438621a2cdf83b462077ff4c9afdb1e6848e118bd4c16"} Jul 28 00:16:23.710465 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.710314 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" event={"ID":"335792df-10dd-4882-81f1-6112414e6631","Type":"ContainerStarted","Data":"13e05cb8aa2f2c2c0b2f753e03e998795328701ef7defcb2b6399182ae1a48bc"} Jul 28 00:16:23.710465 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.710345 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:23.712046 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.712008 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" event={"ID":"e9f224be-5549-477b-ac38-522232c4c761","Type":"ContainerStarted","Data":"f3be4bcc0fdc61d34240548b94e0c3c65a1f0bce5fba44f21dea81f19b7a1827"} Jul 28 00:16:23.712180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.712061 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" event={"ID":"e9f224be-5549-477b-ac38-522232c4c761","Type":"ContainerStarted","Data":"73bf5bdb4bb3d500bfedb7a4ac2d84367c8354501c0295b7dc624ddd461cebea"} Jul 28 00:16:23.712180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.712093 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:23.713732 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.713698 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" event={"ID":"858d61d2-540f-49d6-864a-664fc1a72708","Type":"ContainerStarted","Data":"effefece0f038c77ed2d713fe366dbfe87a03f89972947cbe0e9517434735638"} Jul 28 00:16:23.713857 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.713756 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" event={"ID":"858d61d2-540f-49d6-864a-664fc1a72708","Type":"ContainerStarted","Data":"9d875a03700d651b4aa2254c2ef069dc31359a83893a92cfb5effd24326487f5"} Jul 28 00:16:23.713857 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.713785 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:23.723535 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.723412 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" podStartSLOduration=0.723395478 podStartE2EDuration="723.395478ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:23.723363827 +0000 UTC m=+2383.647572729" watchObservedRunningTime="2026-07-28 00:16:23.723395478 +0000 UTC m=+2383.647604358" Jul 28 00:16:23.737285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.737248 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" podStartSLOduration=1.195284288 podStartE2EDuration="1.73723513s" totalImagesPullingTime="541.950842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:23.736936254 +0000 UTC m=+2383.661145132" watchObservedRunningTime="2026-07-28 00:16:23.73723513 +0000 UTC m=+2383.661444008" Jul 28 00:16:23.751358 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:23.751317 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" podStartSLOduration=1.7513050319999999 podStartE2EDuration="1.751305032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:23.749756117 +0000 UTC m=+2383.673964997" watchObservedRunningTime="2026-07-28 00:16:23.751305032 +0000 UTC m=+2383.675513909" Jul 28 00:16:26.227664 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:26.227632 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:26.328243 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:26.328183 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:26.735355 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:26.735317 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:27.088252 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:27.088215 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:27.395131 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:27.395034 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:27.502240 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:27.502211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:29.730473 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:29.730437 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:16:30.237073 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.237031 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:30.323398 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.323351 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:30.327452 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.327429 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx"] Jul 28 00:16:30.331178 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.331158 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.333620 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.333599 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-6e175a-serving-cert\"" Jul 28 00:16:30.334888 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.334865 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842e7ea4-32f1-4418-adf3-498ac4383318-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.334990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.334892 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842e7ea4-32f1-4418-adf3-498ac4383318-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.334990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.334920 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgb5m\" (UniqueName: \"kubernetes.io/projected/842e7ea4-32f1-4418-adf3-498ac4383318-kube-api-access-lgb5m\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.346518 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.346481 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx"] Jul 28 00:16:30.395117 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.395077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:30.436039 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.436008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842e7ea4-32f1-4418-adf3-498ac4383318-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.436039 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.436040 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842e7ea4-32f1-4418-adf3-498ac4383318-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.436226 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.436068 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgb5m\" (UniqueName: \"kubernetes.io/projected/842e7ea4-32f1-4418-adf3-498ac4383318-kube-api-access-lgb5m\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.436690 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.436662 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842e7ea4-32f1-4418-adf3-498ac4383318-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.438484 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.438459 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842e7ea4-32f1-4418-adf3-498ac4383318-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.450800 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.450780 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgb5m\" (UniqueName: \"kubernetes.io/projected/842e7ea4-32f1-4418-adf3-498ac4383318-kube-api-access-lgb5m\") pod \"maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.498170 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.498146 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:30.642609 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.642571 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:30.778034 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.778007 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx"] Jul 28 00:16:30.779692 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:30.779666 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842e7ea4_32f1_4418_adf3_498ac4383318.slice/crio-fba2e78249207b32ca25e84f5c1962e2c4a95df791013dfa385b1c6d107c97bb WatchSource:0}: Error finding container fba2e78249207b32ca25e84f5c1962e2c4a95df791013dfa385b1c6d107c97bb: Status 404 returned error can't find the container with id fba2e78249207b32ca25e84f5c1962e2c4a95df791013dfa385b1c6d107c97bb Jul 28 00:16:30.912835 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:30.912794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:31.029712 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.029627 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:31.046190 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.046161 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms"] Jul 28 00:16:31.050939 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.050923 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.054244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.054228 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-6e175a\"" Jul 28 00:16:31.059908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.059884 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k"] Jul 28 00:16:31.063323 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.063306 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.065895 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.065873 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-6e175a-dockercfg-6x2w6\"" Jul 28 00:16:31.069050 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.069029 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms"] Jul 28 00:16:31.079242 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.079218 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k"] Jul 28 00:16:31.142342 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.142306 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.142477 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.142354 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lsc\" (UniqueName: \"kubernetes.io/projected/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-kube-api-access-j2lsc\") pod \"payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.243082 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.243051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc28t\" (UniqueName: \"kubernetes.io/projected/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-kube-api-access-sc28t\") pod \"payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.243247 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.243101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.243247 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.243164 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lsc\" (UniqueName: \"kubernetes.io/projected/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-kube-api-access-j2lsc\") pod \"payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.243247 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.243219 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.243864 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.243842 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.251071 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.251054 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lsc\" (UniqueName: \"kubernetes.io/projected/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-kube-api-access-j2lsc\") pod \"payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.343732 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.343705 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc28t\" (UniqueName: \"kubernetes.io/projected/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-kube-api-access-sc28t\") pod \"payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.343847 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.343793 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.344397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.344371 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.352844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.352823 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc28t\" (UniqueName: \"kubernetes.io/projected/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-kube-api-access-sc28t\") pod \"payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.361654 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.361633 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:31.373741 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.373715 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.542897 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.542872 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k"] Jul 28 00:16:31.544463 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:31.544432 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bf45cf_c03b_4e10_8f45_c46b4d25cdbe.slice/crio-8cbc35199322410cb0217079bd70d0d2c524cbea42a8b330cc1c6f95b1f5323b WatchSource:0}: Error finding container 8cbc35199322410cb0217079bd70d0d2c524cbea42a8b330cc1c6f95b1f5323b: Status 404 returned error can't find the container with id 8cbc35199322410cb0217079bd70d0d2c524cbea42a8b330cc1c6f95b1f5323b Jul 28 00:16:31.717003 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.716969 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms"] Jul 28 00:16:31.717806 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:31.717778 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a18cc2b_4f56_48d2_8cc5_dace579cfdad.slice/crio-dab9efb1e129667eb6887d8207fd6c80dbf11c58d2b6b975b407f4b278d199f2 WatchSource:0}: Error finding container dab9efb1e129667eb6887d8207fd6c80dbf11c58d2b6b975b407f4b278d199f2: Status 404 returned error can't find the container with id dab9efb1e129667eb6887d8207fd6c80dbf11c58d2b6b975b407f4b278d199f2 Jul 28 00:16:31.763567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.763539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" event={"ID":"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe","Type":"ContainerStarted","Data":"40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c"} Jul 28 00:16:31.763688 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.763600 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" event={"ID":"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe","Type":"ContainerStarted","Data":"8cbc35199322410cb0217079bd70d0d2c524cbea42a8b330cc1c6f95b1f5323b"} Jul 28 00:16:31.763688 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.763632 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:31.764825 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.764805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" event={"ID":"0a18cc2b-4f56-48d2-8cc5-dace579cfdad","Type":"ContainerStarted","Data":"dab9efb1e129667eb6887d8207fd6c80dbf11c58d2b6b975b407f4b278d199f2"} Jul 28 00:16:31.766223 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.766199 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" event={"ID":"842e7ea4-32f1-4418-adf3-498ac4383318","Type":"ContainerStarted","Data":"5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5"} Jul 28 00:16:31.766223 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.766248 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" event={"ID":"842e7ea4-32f1-4418-adf3-498ac4383318","Type":"ContainerStarted","Data":"fba2e78249207b32ca25e84f5c1962e2c4a95df791013dfa385b1c6d107c97bb"} Jul 28 00:16:31.766421 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.766281 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:31.782908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.782859 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" podStartSLOduration=0.78283994 podStartE2EDuration="782.83994ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:31.779927125 +0000 UTC m=+2391.704136003" watchObservedRunningTime="2026-07-28 00:16:31.78283994 +0000 UTC m=+2391.707048821" Jul 28 00:16:31.799699 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:31.799610 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" podStartSLOduration=1.330381137 podStartE2EDuration="1.79958957s" totalImagesPullingTime="469.208433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:31.796016645 +0000 UTC m=+2391.720225524" watchObservedRunningTime="2026-07-28 00:16:31.79958957 +0000 UTC m=+2391.723798456" Jul 28 00:16:32.772936 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:32.772903 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" event={"ID":"0a18cc2b-4f56-48d2-8cc5-dace579cfdad","Type":"ContainerStarted","Data":"4b622634ebb8ef959b407ee21a805e2f6305f737bcf1594ea6970a079744b876"} Jul 28 00:16:32.788464 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:32.788409 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" podStartSLOduration=1.788393645 podStartE2EDuration="1.788393645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:32.786797274 +0000 UTC m=+2392.711006151" watchObservedRunningTime="2026-07-28 00:16:32.788393645 +0000 UTC m=+2392.712602501" Jul 28 00:16:33.778508 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:33.778469 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:33.914306 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:33.914264 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:34.016782 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:34.016749 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:34.726073 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:34.726032 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" podUID="858d61d2-540f-49d6-864a-664fc1a72708" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.94:9004: connect: connection refused" Jul 28 00:16:34.726599 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:34.726578 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:16:34.737747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:34.737717 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:37.778340 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:37.778313 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:16:41.073152 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:41.073120 2555 scope.go:117] "RemoveContainer" containerID="31817204b883e839080f65125c606e9f7b4c17f773d58e086df6c7633b60ddee" Jul 28 00:16:42.775371 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:42.775339 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:16:44.725940 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:44.725907 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:16:44.787304 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:44.787273 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:16:51.839874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.839840 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v"] Jul 28 00:16:51.843476 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.843456 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:51.845827 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.845795 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"default-dockercfg-7wvks\"" Jul 28 00:16:51.845827 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.845817 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"kube-root-ca.crt\"" Jul 28 00:16:51.846677 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.846655 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"auth-test-model-328ef1-kserve-self-signed-certs\"" Jul 28 00:16:51.846776 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.846705 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"openshift-service-ca.crt\"" Jul 28 00:16:51.851980 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.851961 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v"] Jul 28 00:16:51.867009 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.866979 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:16:51.915790 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.915767 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5mx\" (UniqueName: \"kubernetes.io/projected/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-kube-api-access-6x5mx\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:51.915891 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.915797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-model-cache\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:51.915940 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.915887 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-home\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:51.915940 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.915913 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tls-certs\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:51.915940 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.915938 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-dshm\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:51.916040 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:51.915954 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tmp-dir\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017379 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017335 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-home\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017379 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017384 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tls-certs\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017638 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-dshm\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017638 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tmp-dir\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017638 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017454 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5mx\" (UniqueName: \"kubernetes.io/projected/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-kube-api-access-6x5mx\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017638 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017480 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-model-cache\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.017935 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.017908 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-home\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.018113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.018084 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tmp-dir\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.018227 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.018152 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-model-cache\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.019639 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.019620 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-dshm\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.019903 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.019884 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tls-certs\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.024323 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.024301 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5mx\" (UniqueName: \"kubernetes.io/projected/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-kube-api-access-6x5mx\") pod \"auth-test-model-328ef1-kserve-7df9f96467-9tx9v\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.154044 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.153967 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:16:52.291620 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.291591 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v"] Jul 28 00:16:52.294037 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:16:52.293998 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35b0d840_757f_4733_8ede_bcd7ffb0f2ac.slice/crio-f17e26bdb33b7c50c4a86e91490a13bef9d8ccffb9594aa094d015b5b6c15716 WatchSource:0}: Error finding container f17e26bdb33b7c50c4a86e91490a13bef9d8ccffb9594aa094d015b5b6c15716: Status 404 returned error can't find the container with id f17e26bdb33b7c50c4a86e91490a13bef9d8ccffb9594aa094d015b5b6c15716 Jul 28 00:16:52.880027 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.879993 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" event={"ID":"35b0d840-757f-4733-8ede-bcd7ffb0f2ac","Type":"ContainerStarted","Data":"e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e"} Jul 28 00:16:52.880438 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.880038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" event={"ID":"35b0d840-757f-4733-8ede-bcd7ffb0f2ac","Type":"ContainerStarted","Data":"f17e26bdb33b7c50c4a86e91490a13bef9d8ccffb9594aa094d015b5b6c15716"} Jul 28 00:16:52.896333 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:16:52.896290 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" podStartSLOduration=1.896276318 podStartE2EDuration="1.896276318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:16:52.893931445 +0000 UTC m=+2412.818140322" watchObservedRunningTime="2026-07-28 00:16:52.896276318 +0000 UTC m=+2412.820485196" Jul 28 00:17:02.154676 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:02.154624 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:17:02.154676 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:02.154689 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:17:02.167254 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:02.167224 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:17:02.943108 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:02.943027 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:17:23.135997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.135960 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw"] Jul 28 00:17:23.139740 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.139718 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.142104 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.142081 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"kube-root-ca.crt\"" Jul 28 00:17:23.142642 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.142620 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"openshift-service-ca.crt\"" Jul 28 00:17:23.142762 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.142625 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"default-dockercfg-ckwtx\"" Jul 28 00:17:23.142762 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.142754 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"auth-test-model-6e175a-kserve-self-signed-certs\"" Jul 28 00:17:23.150234 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.150207 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw"] Jul 28 00:17:23.192540 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.192480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-model-cache\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.192709 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.192634 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-dshm\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.192709 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.192661 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tls-certs\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.192786 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.192734 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-home\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.192821 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.192797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tmp-dir\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.192821 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.192815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h266\" (UniqueName: \"kubernetes.io/projected/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-kube-api-access-4h266\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.234523 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.234474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:17:23.293696 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.293664 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-dshm\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.293844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.293701 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tls-certs\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.293844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.293737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-home\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.293844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.293779 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tmp-dir\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.293844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.293795 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h266\" (UniqueName: \"kubernetes.io/projected/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-kube-api-access-4h266\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.293844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.293816 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-model-cache\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.294192 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.294168 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tmp-dir\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.294265 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.294221 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-home\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.294265 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.294251 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-model-cache\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.295983 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.295963 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-dshm\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.296267 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.296245 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tls-certs\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.304171 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.304147 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h266\" (UniqueName: \"kubernetes.io/projected/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-kube-api-access-4h266\") pod \"auth-test-model-6e175a-kserve-85bd8b785f-tkhsw\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.449605 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.449529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:23.592995 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:23.592963 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw"] Jul 28 00:17:23.594956 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:17:23.594916 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod460cd6cd_1b4f_4ae2_be80_8f1b2119cc9a.slice/crio-9c96459a162c6e39edea2c37e25ddff9553909e0cce3899c7124675863de508d WatchSource:0}: Error finding container 9c96459a162c6e39edea2c37e25ddff9553909e0cce3899c7124675863de508d: Status 404 returned error can't find the container with id 9c96459a162c6e39edea2c37e25ddff9553909e0cce3899c7124675863de508d Jul 28 00:17:24.039283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:24.039247 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" event={"ID":"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a","Type":"ContainerStarted","Data":"406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9"} Jul 28 00:17:24.039459 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:24.039294 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" event={"ID":"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a","Type":"ContainerStarted","Data":"9c96459a162c6e39edea2c37e25ddff9553909e0cce3899c7124675863de508d"} Jul 28 00:17:24.054229 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:24.054182 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" podStartSLOduration=1.054165199 podStartE2EDuration="1.054165199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:17:24.053608149 +0000 UTC m=+2443.977817093" watchObservedRunningTime="2026-07-28 00:17:24.054165199 +0000 UTC m=+2443.978374076" Jul 28 00:17:33.450542 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:33.450484 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:33.450961 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:33.450557 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:33.463016 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:33.462993 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:17:34.105142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:17:34.105111 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:19:53.044855 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:19:53.044818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:19:53.134978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:19:53.134935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:20:41.138965 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:20:41.138931 2555 scope.go:117] "RemoveContainer" containerID="bdb05fff9b9f97791bd8d794b48b0540909c2592f02b62e5283c1af2e1eed462" Jul 28 00:21:19.226628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:19.226574 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:21:20.926688 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:20.926648 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:21:41.160326 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:41.160238 2555 scope.go:117] "RemoveContainer" containerID="34eb5d921ecf9d3d070fa52485f0d308142917707bf963f35a9f671ac4cd31ce" Jul 28 00:21:46.619247 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:46.619210 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v"] Jul 28 00:21:46.639744 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:46.639710 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:21:47.645186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:47.645144 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw"] Jul 28 00:21:47.661361 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:21:47.661324 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:22:01.055347 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.055292 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:22:01.640715 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.640663 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" podUID="35b0d840-757f-4733-8ede-bcd7ffb0f2ac" containerName="main" containerID="cri-o://e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e" gracePeriod=45 Jul 28 00:22:01.912705 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.912677 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:22:01.974919 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.974880 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tmp-dir\") pod \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " Jul 28 00:22:01.975097 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.974978 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-home\") pod \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " Jul 28 00:22:01.975097 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975058 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tls-certs\") pod \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " Jul 28 00:22:01.975171 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975102 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-dshm\") pod \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " Jul 28 00:22:01.975171 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975139 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-model-cache\") pod \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " Jul 28 00:22:01.975234 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975168 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35b0d840-757f-4733-8ede-bcd7ffb0f2ac" (UID: "35b0d840-757f-4733-8ede-bcd7ffb0f2ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:01.975234 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975181 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-home" (OuterVolumeSpecName: "home") pod "35b0d840-757f-4733-8ede-bcd7ffb0f2ac" (UID: "35b0d840-757f-4733-8ede-bcd7ffb0f2ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:01.975234 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975185 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x5mx\" (UniqueName: \"kubernetes.io/projected/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-kube-api-access-6x5mx\") pod \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\" (UID: \"35b0d840-757f-4733-8ede-bcd7ffb0f2ac\") " Jul 28 00:22:01.975370 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975354 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "35b0d840-757f-4733-8ede-bcd7ffb0f2ac" (UID: "35b0d840-757f-4733-8ede-bcd7ffb0f2ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:01.975560 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975517 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:01.975560 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975536 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:01.975560 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.975545 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:01.977371 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.977342 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35b0d840-757f-4733-8ede-bcd7ffb0f2ac" (UID: "35b0d840-757f-4733-8ede-bcd7ffb0f2ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:22:01.977371 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.977365 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-dshm" (OuterVolumeSpecName: "dshm") pod "35b0d840-757f-4733-8ede-bcd7ffb0f2ac" (UID: "35b0d840-757f-4733-8ede-bcd7ffb0f2ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:01.977514 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:01.977388 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-kube-api-access-6x5mx" (OuterVolumeSpecName: "kube-api-access-6x5mx") pod "35b0d840-757f-4733-8ede-bcd7ffb0f2ac" (UID: "35b0d840-757f-4733-8ede-bcd7ffb0f2ac"). InnerVolumeSpecName "kube-api-access-6x5mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:22:02.076868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.076828 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:02.076868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.076860 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:02.076868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.076870 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5mx\" (UniqueName: \"kubernetes.io/projected/35b0d840-757f-4733-8ede-bcd7ffb0f2ac-kube-api-access-6x5mx\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:02.463549 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.463517 2555 generic.go:356] "Generic (PLEG): container finished" podID="35b0d840-757f-4733-8ede-bcd7ffb0f2ac" containerID="e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e" exitCode=0 Jul 28 00:22:02.463728 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.463578 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" Jul 28 00:22:02.463728 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.463596 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" event={"ID":"35b0d840-757f-4733-8ede-bcd7ffb0f2ac","Type":"ContainerDied","Data":"e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e"} Jul 28 00:22:02.463728 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.463644 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v" event={"ID":"35b0d840-757f-4733-8ede-bcd7ffb0f2ac","Type":"ContainerDied","Data":"f17e26bdb33b7c50c4a86e91490a13bef9d8ccffb9594aa094d015b5b6c15716"} Jul 28 00:22:02.463728 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.463666 2555 scope.go:117] "RemoveContainer" containerID="e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e" Jul 28 00:22:02.472831 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.472810 2555 scope.go:117] "RemoveContainer" containerID="e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e" Jul 28 00:22:02.473094 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:22:02.473075 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e\": container with ID starting with e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e not found: ID does not exist" containerID="e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e" Jul 28 00:22:02.473146 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.473105 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e"} err="failed to get container status \"e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e\": rpc error: code = NotFound desc = could not find container \"e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e\": container with ID starting with e90675c3390d939f7e5f343d94d4e51a7d7b0667a1c7fed9da7de00e51f7870e not found: ID does not exist" Jul 28 00:22:02.487112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.487084 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v"] Jul 28 00:22:02.490767 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.490743 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/auth-test-model-328ef1-kserve-7df9f96467-9tx9v"] Jul 28 00:22:02.566374 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.566336 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b0d840-757f-4733-8ede-bcd7ffb0f2ac" path="/var/lib/kubelet/pods/35b0d840-757f-4733-8ede-bcd7ffb0f2ac/volumes" Jul 28 00:22:02.664841 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.664797 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" podUID="460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" containerName="main" containerID="cri-o://406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9" gracePeriod=45 Jul 28 00:22:02.926601 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.926577 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:22:02.984696 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.984664 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tls-certs\") pod \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " Jul 28 00:22:02.984885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.984706 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-dshm\") pod \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " Jul 28 00:22:02.984885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.984735 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-home\") pod \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " Jul 28 00:22:02.984885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.984773 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h266\" (UniqueName: \"kubernetes.io/projected/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-kube-api-access-4h266\") pod \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " Jul 28 00:22:02.984885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.984822 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-model-cache\") pod \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " Jul 28 00:22:02.984885 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.984864 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tmp-dir\") pod \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\" (UID: \"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a\") " Jul 28 00:22:02.985221 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.985080 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-home" (OuterVolumeSpecName: "home") pod "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" (UID: "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:02.985221 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.985174 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" (UID: "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:02.985349 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.985311 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" (UID: "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:02.986939 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.986917 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" (UID: "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:22:02.987275 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.987251 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-dshm" (OuterVolumeSpecName: "dshm") pod "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" (UID: "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:22:02.987372 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:02.987279 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-kube-api-access-4h266" (OuterVolumeSpecName: "kube-api-access-4h266") pod "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" (UID: "460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a"). InnerVolumeSpecName "kube-api-access-4h266". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:22:03.086224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.086143 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:03.086224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.086172 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:03.086224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.086181 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:03.086224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.086189 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:03.086224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.086197 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:03.086224 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.086205 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h266\" (UniqueName: \"kubernetes.io/projected/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a-kube-api-access-4h266\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:22:03.469432 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.469335 2555 generic.go:356] "Generic (PLEG): container finished" podID="460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" containerID="406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9" exitCode=0 Jul 28 00:22:03.469432 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.469381 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" Jul 28 00:22:03.469432 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.469423 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" event={"ID":"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a","Type":"ContainerDied","Data":"406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9"} Jul 28 00:22:03.469734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.469463 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw" event={"ID":"460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a","Type":"ContainerDied","Data":"9c96459a162c6e39edea2c37e25ddff9553909e0cce3899c7124675863de508d"} Jul 28 00:22:03.469734 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.469483 2555 scope.go:117] "RemoveContainer" containerID="406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9" Jul 28 00:22:03.479319 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.479294 2555 scope.go:117] "RemoveContainer" containerID="406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9" Jul 28 00:22:03.479614 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:22:03.479595 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9\": container with ID starting with 406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9 not found: ID does not exist" containerID="406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9" Jul 28 00:22:03.479679 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.479624 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9"} err="failed to get container status \"406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9\": rpc error: code = NotFound desc = could not find container \"406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9\": container with ID starting with 406a2901f1e559c5d3ea332edec2376bff13eab5e3e0f4c91bcfb5dcfbfd77a9 not found: ID does not exist" Jul 28 00:22:03.489946 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.489921 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw"] Jul 28 00:22:03.493406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:03.493385 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/auth-test-model-6e175a-kserve-85bd8b785f-tkhsw"] Jul 28 00:22:04.565392 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:04.565358 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" path="/var/lib/kubelet/pods/460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a/volumes" Jul 28 00:22:15.568767 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:15.568732 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:22:19.859589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.859555 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55"] Jul 28 00:22:19.860091 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.860073 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" containerName="main" Jul 28 00:22:19.860091 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.860093 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" containerName="main" Jul 28 00:22:19.860182 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.860115 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b0d840-757f-4733-8ede-bcd7ffb0f2ac" containerName="main" Jul 28 00:22:19.860182 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.860120 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b0d840-757f-4733-8ede-bcd7ffb0f2ac" containerName="main" Jul 28 00:22:19.860245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.860189 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="460cd6cd-1b4f-4ae2-be80-8f1b2119cc9a" containerName="main" Jul 28 00:22:19.860245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.860196 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b0d840-757f-4733-8ede-bcd7ffb0f2ac" containerName="main" Jul 28 00:22:19.863291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.863275 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:19.866313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.866287 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"default-dockercfg-ckwtx\"" Jul 28 00:22:19.866431 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.866362 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"sub-test-model-6e175a-kserve-self-signed-certs\"" Jul 28 00:22:19.866431 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.866362 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"kube-root-ca.crt\"" Jul 28 00:22:19.866431 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.866421 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"openshift-service-ca.crt\"" Jul 28 00:22:19.875481 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.875455 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55"] Jul 28 00:22:19.888407 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.888353 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:22:19.934244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.934216 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-tmp-dir\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:19.934428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.934248 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8nb\" (UniqueName: \"kubernetes.io/projected/9367426a-6043-46bf-8394-1b5ae3cc8da8-kube-api-access-dm8nb\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:19.934428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.934329 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9367426a-6043-46bf-8394-1b5ae3cc8da8-tls-certs\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:19.934428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.934406 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-home\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:19.934617 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.934440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-dshm\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:19.934617 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:19.934468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-model-cache\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.035887 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.035846 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-dshm\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.035887 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.035894 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-model-cache\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.035955 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-tmp-dir\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.035980 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8nb\" (UniqueName: \"kubernetes.io/projected/9367426a-6043-46bf-8394-1b5ae3cc8da8-kube-api-access-dm8nb\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.036047 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9367426a-6043-46bf-8394-1b5ae3cc8da8-tls-certs\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.036099 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-home\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036436 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.036410 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-tmp-dir\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036534 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.036439 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-model-cache\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.036534 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.036476 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-home\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.038244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.038226 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-dshm\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.038416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.038398 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9367426a-6043-46bf-8394-1b5ae3cc8da8-tls-certs\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.043768 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.043745 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8nb\" (UniqueName: \"kubernetes.io/projected/9367426a-6043-46bf-8394-1b5ae3cc8da8-kube-api-access-dm8nb\") pod \"sub-test-model-6e175a-kserve-dd7b74897-zhr55\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.172179 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.172093 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:20.316176 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.316150 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55"] Jul 28 00:22:20.318376 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:22:20.318349 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9367426a_6043_46bf_8394_1b5ae3cc8da8.slice/crio-48637134688ee9cc9dd27438c3d87f5430ce031d70114952c05bf9f05831aa3f WatchSource:0}: Error finding container 48637134688ee9cc9dd27438c3d87f5430ce031d70114952c05bf9f05831aa3f: Status 404 returned error can't find the container with id 48637134688ee9cc9dd27438c3d87f5430ce031d70114952c05bf9f05831aa3f Jul 28 00:22:20.556284 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.556243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" event={"ID":"9367426a-6043-46bf-8394-1b5ae3cc8da8","Type":"ContainerStarted","Data":"60baeedba64b563c9663b64ae276e9f44dec5300165a091ad795dafe69d647db"} Jul 28 00:22:20.556477 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.556298 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" event={"ID":"9367426a-6043-46bf-8394-1b5ae3cc8da8","Type":"ContainerStarted","Data":"48637134688ee9cc9dd27438c3d87f5430ce031d70114952c05bf9f05831aa3f"} Jul 28 00:22:20.573460 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:20.573408 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" podStartSLOduration=1.573391889 podStartE2EDuration="1.573391889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:22:20.57174664 +0000 UTC m=+2740.495955522" watchObservedRunningTime="2026-07-28 00:22:20.573391889 +0000 UTC m=+2740.497600767" Jul 28 00:22:30.173104 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:30.173054 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:30.173104 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:30.173115 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:30.185719 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:30.185693 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:30.617204 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:30.617177 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:22:41.181580 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:41.181549 2555 scope.go:117] "RemoveContainer" containerID="7fb76e825404157c9ee91d69e9d5ff7c375e71cf4750a116ba889e90b8c35b6e" Jul 28 00:22:45.623636 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:22:45.623600 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:23:35.045755 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:35.045671 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:23:35.822671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:35.822636 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55"] Jul 28 00:23:35.844027 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:35.843996 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:23:50.842532 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:50.842472 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" podUID="9367426a-6043-46bf-8394-1b5ae3cc8da8" containerName="main" containerID="cri-o://60baeedba64b563c9663b64ae276e9f44dec5300165a091ad795dafe69d647db" gracePeriod=45 Jul 28 00:23:51.016601 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.016567 2555 generic.go:356] "Generic (PLEG): container finished" podID="9367426a-6043-46bf-8394-1b5ae3cc8da8" containerID="60baeedba64b563c9663b64ae276e9f44dec5300165a091ad795dafe69d647db" exitCode=0 Jul 28 00:23:51.016737 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.016634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" event={"ID":"9367426a-6043-46bf-8394-1b5ae3cc8da8","Type":"ContainerDied","Data":"60baeedba64b563c9663b64ae276e9f44dec5300165a091ad795dafe69d647db"} Jul 28 00:23:51.095067 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.095012 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:23:51.242338 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.242295 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff"] Jul 28 00:23:51.244111 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.244082 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9367426a-6043-46bf-8394-1b5ae3cc8da8" containerName="main" Jul 28 00:23:51.244111 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.244112 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9367426a-6043-46bf-8394-1b5ae3cc8da8" containerName="main" Jul 28 00:23:51.244417 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.244395 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9367426a-6043-46bf-8394-1b5ae3cc8da8" containerName="main" Jul 28 00:23:51.248460 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.248435 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.251609 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.251583 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"kube-root-ca.crt\"" Jul 28 00:23:51.251759 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.251589 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"openshift-service-ca.crt\"" Jul 28 00:23:51.251759 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.251636 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"rate-test-model-328ef1-kserve-self-signed-certs\"" Jul 28 00:23:51.251759 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.251615 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-328ef1\"/\"default-dockercfg-7wvks\"" Jul 28 00:23:51.253233 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.253211 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-tmp-dir\") pod \"9367426a-6043-46bf-8394-1b5ae3cc8da8\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " Jul 28 00:23:51.253972 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.253409 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9367426a-6043-46bf-8394-1b5ae3cc8da8" (UID: "9367426a-6043-46bf-8394-1b5ae3cc8da8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:23:51.253972 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.253816 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9367426a-6043-46bf-8394-1b5ae3cc8da8-tls-certs\") pod \"9367426a-6043-46bf-8394-1b5ae3cc8da8\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " Jul 28 00:23:51.253972 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.253931 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm8nb\" (UniqueName: \"kubernetes.io/projected/9367426a-6043-46bf-8394-1b5ae3cc8da8-kube-api-access-dm8nb\") pod \"9367426a-6043-46bf-8394-1b5ae3cc8da8\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " Jul 28 00:23:51.254160 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.253968 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-home\") pod \"9367426a-6043-46bf-8394-1b5ae3cc8da8\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " Jul 28 00:23:51.254160 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.254017 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-model-cache\") pod \"9367426a-6043-46bf-8394-1b5ae3cc8da8\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " Jul 28 00:23:51.254160 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.254039 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-dshm\") pod \"9367426a-6043-46bf-8394-1b5ae3cc8da8\" (UID: \"9367426a-6043-46bf-8394-1b5ae3cc8da8\") " Jul 28 00:23:51.254291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.254262 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-home" (OuterVolumeSpecName: "home") pod "9367426a-6043-46bf-8394-1b5ae3cc8da8" (UID: "9367426a-6043-46bf-8394-1b5ae3cc8da8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:23:51.254524 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.254478 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-model-cache" (OuterVolumeSpecName: "model-cache") pod "9367426a-6043-46bf-8394-1b5ae3cc8da8" (UID: "9367426a-6043-46bf-8394-1b5ae3cc8da8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:23:51.254950 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.254921 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:23:51.255100 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.255080 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:23:51.255186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.255102 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:23:51.255186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.255055 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff"] Jul 28 00:23:51.256948 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.256926 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-dshm" (OuterVolumeSpecName: "dshm") pod "9367426a-6043-46bf-8394-1b5ae3cc8da8" (UID: "9367426a-6043-46bf-8394-1b5ae3cc8da8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:23:51.257084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.257004 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9367426a-6043-46bf-8394-1b5ae3cc8da8-kube-api-access-dm8nb" (OuterVolumeSpecName: "kube-api-access-dm8nb") pod "9367426a-6043-46bf-8394-1b5ae3cc8da8" (UID: "9367426a-6043-46bf-8394-1b5ae3cc8da8"). InnerVolumeSpecName "kube-api-access-dm8nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:23:51.257188 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.257156 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9367426a-6043-46bf-8394-1b5ae3cc8da8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9367426a-6043-46bf-8394-1b5ae3cc8da8" (UID: "9367426a-6043-46bf-8394-1b5ae3cc8da8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:23:51.269747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.269725 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:23:51.355529 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355446 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93000e3a-bf38-4568-bd65-1cf07926082c-tls-certs\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.355529 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355478 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsnm\" (UniqueName: \"kubernetes.io/projected/93000e3a-bf38-4568-bd65-1cf07926082c-kube-api-access-frsnm\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.355529 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355524 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-tmp-dir\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.355695 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-dshm\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.355695 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-model-cache\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.355695 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355626 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-home\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.355695 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355690 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9367426a-6043-46bf-8394-1b5ae3cc8da8-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:23:51.355826 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355700 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8nb\" (UniqueName: \"kubernetes.io/projected/9367426a-6043-46bf-8394-1b5ae3cc8da8-kube-api-access-dm8nb\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:23:51.355826 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.355709 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9367426a-6043-46bf-8394-1b5ae3cc8da8-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:23:51.456155 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456124 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-home\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93000e3a-bf38-4568-bd65-1cf07926082c-tls-certs\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsnm\" (UniqueName: \"kubernetes.io/projected/93000e3a-bf38-4568-bd65-1cf07926082c-kube-api-access-frsnm\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456231 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-tmp-dir\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456291 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-dshm\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456313 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456311 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-model-cache\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456646 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456618 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-home\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456725 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456657 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-model-cache\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.456725 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.456696 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-tmp-dir\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.458607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.458589 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-dshm\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.458803 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.458785 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93000e3a-bf38-4568-bd65-1cf07926082c-tls-certs\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.463791 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.463766 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsnm\" (UniqueName: \"kubernetes.io/projected/93000e3a-bf38-4568-bd65-1cf07926082c-kube-api-access-frsnm\") pod \"rate-test-model-328ef1-kserve-565495b567-chcff\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.569663 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.569635 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:23:51.706244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:51.706165 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff"] Jul 28 00:23:51.709082 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:23:51.709058 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93000e3a_bf38_4568_bd65_1cf07926082c.slice/crio-106bb1e2c344efe8f8cf80d3ab9113d60ad6587094d2b10938c95221d2cfd37b WatchSource:0}: Error finding container 106bb1e2c344efe8f8cf80d3ab9113d60ad6587094d2b10938c95221d2cfd37b: Status 404 returned error can't find the container with id 106bb1e2c344efe8f8cf80d3ab9113d60ad6587094d2b10938c95221d2cfd37b Jul 28 00:23:52.023965 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.023932 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" Jul 28 00:23:52.023965 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.023954 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55" event={"ID":"9367426a-6043-46bf-8394-1b5ae3cc8da8","Type":"ContainerDied","Data":"48637134688ee9cc9dd27438c3d87f5430ce031d70114952c05bf9f05831aa3f"} Jul 28 00:23:52.024485 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.024011 2555 scope.go:117] "RemoveContainer" containerID="60baeedba64b563c9663b64ae276e9f44dec5300165a091ad795dafe69d647db" Jul 28 00:23:52.025919 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.025892 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" event={"ID":"93000e3a-bf38-4568-bd65-1cf07926082c","Type":"ContainerStarted","Data":"71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e"} Jul 28 00:23:52.026040 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.025925 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" event={"ID":"93000e3a-bf38-4568-bd65-1cf07926082c","Type":"ContainerStarted","Data":"106bb1e2c344efe8f8cf80d3ab9113d60ad6587094d2b10938c95221d2cfd37b"} Jul 28 00:23:52.047327 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.047289 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" podStartSLOduration=1.047274276 podStartE2EDuration="1.047274276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:23:52.04565322 +0000 UTC m=+2831.969862088" watchObservedRunningTime="2026-07-28 00:23:52.047274276 +0000 UTC m=+2831.971483153" Jul 28 00:23:52.059406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.059378 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55"] Jul 28 00:23:52.063216 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.063196 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/sub-test-model-6e175a-kserve-dd7b74897-zhr55"] Jul 28 00:23:52.566514 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:23:52.566461 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9367426a-6043-46bf-8394-1b5ae3cc8da8" path="/var/lib/kubelet/pods/9367426a-6043-46bf-8394-1b5ae3cc8da8/volumes" Jul 28 00:24:01.569769 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:01.569733 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:24:01.569769 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:01.569778 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:24:01.583932 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:01.583902 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:24:02.087280 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:02.087249 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:24:32.844005 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.843927 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp"] Jul 28 00:24:32.853105 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.853077 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:32.856265 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.856228 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"default-dockercfg-ckwtx\"" Jul 28 00:24:32.856798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.856766 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"kube-root-ca.crt\"" Jul 28 00:24:32.856798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.856767 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"openshift-service-ca.crt\"" Jul 28 00:24:32.856991 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.856813 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6e175a\"/\"rate-test-model-6e175a-kserve-self-signed-certs\"" Jul 28 00:24:32.857542 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.857519 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp"] Jul 28 00:24:32.865479 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.865458 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:24:32.920689 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.920657 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-dshm\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:32.920689 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.920688 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tls-certs\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:32.920892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.920781 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gpj\" (UniqueName: \"kubernetes.io/projected/e7566239-a78e-4f7b-91f4-de5aa5bdca75-kube-api-access-97gpj\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:32.920892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.920809 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tmp-dir\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:32.920892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.920826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-model-cache\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:32.920892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:32.920857 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-home\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.021904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.021858 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tmp-dir\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.021904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.021907 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-model-cache\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.021953 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-home\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.021976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-dshm\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.021997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tls-certs\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.022098 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gpj\" (UniqueName: \"kubernetes.io/projected/e7566239-a78e-4f7b-91f4-de5aa5bdca75-kube-api-access-97gpj\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022373 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.022312 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tmp-dir\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022373 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.022356 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-home\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.022469 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.022372 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-model-cache\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.024373 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.024347 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-dshm\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.024585 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.024566 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tls-certs\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.029140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.029116 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gpj\" (UniqueName: \"kubernetes.io/projected/e7566239-a78e-4f7b-91f4-de5aa5bdca75-kube-api-access-97gpj\") pod \"rate-test-model-6e175a-kserve-689897b4f9-scmjp\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.164749 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.164664 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:33.299665 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:33.299639 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp"] Jul 28 00:24:33.302089 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:24:33.302059 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7566239_a78e_4f7b_91f4_de5aa5bdca75.slice/crio-be3072f57eacae365acc39a377e252534e86bdbc8de470979004fbf2e31527f8 WatchSource:0}: Error finding container be3072f57eacae365acc39a377e252534e86bdbc8de470979004fbf2e31527f8: Status 404 returned error can't find the container with id be3072f57eacae365acc39a377e252534e86bdbc8de470979004fbf2e31527f8 Jul 28 00:24:34.236120 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:34.236084 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" event={"ID":"e7566239-a78e-4f7b-91f4-de5aa5bdca75","Type":"ContainerStarted","Data":"be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3"} Jul 28 00:24:34.236631 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:34.236134 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" event={"ID":"e7566239-a78e-4f7b-91f4-de5aa5bdca75","Type":"ContainerStarted","Data":"be3072f57eacae365acc39a377e252534e86bdbc8de470979004fbf2e31527f8"} Jul 28 00:24:34.252884 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:34.252835 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" podStartSLOduration=2.252818618 podStartE2EDuration="2.252818618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:24:34.250175027 +0000 UTC m=+2874.174383905" watchObservedRunningTime="2026-07-28 00:24:34.252818618 +0000 UTC m=+2874.177027498" Jul 28 00:24:43.165448 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:43.165408 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:43.165878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:43.165461 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:43.177954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:43.177930 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:43.298786 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:43.298756 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:24:59.879709 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:24:59.879674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:25:42.785706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:42.785671 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff"] Jul 28 00:25:42.804573 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:42.804540 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:25:43.638917 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:43.638878 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp"] Jul 28 00:25:43.663015 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:43.662985 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:25:47.473982 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:47.473947 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:25:47.529868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:47.529835 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:25:57.804628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:57.804585 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" podUID="93000e3a-bf38-4568-bd65-1cf07926082c" containerName="main" containerID="cri-o://71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e" gracePeriod=45 Jul 28 00:25:58.064110 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.064046 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:25:58.185200 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185156 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-home\") pod \"93000e3a-bf38-4568-bd65-1cf07926082c\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " Jul 28 00:25:58.185388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185221 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93000e3a-bf38-4568-bd65-1cf07926082c-tls-certs\") pod \"93000e3a-bf38-4568-bd65-1cf07926082c\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " Jul 28 00:25:58.185388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185240 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-tmp-dir\") pod \"93000e3a-bf38-4568-bd65-1cf07926082c\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " Jul 28 00:25:58.185388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185259 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frsnm\" (UniqueName: \"kubernetes.io/projected/93000e3a-bf38-4568-bd65-1cf07926082c-kube-api-access-frsnm\") pod \"93000e3a-bf38-4568-bd65-1cf07926082c\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " Jul 28 00:25:58.185388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185277 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-dshm\") pod \"93000e3a-bf38-4568-bd65-1cf07926082c\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " Jul 28 00:25:58.185388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185291 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-model-cache\") pod \"93000e3a-bf38-4568-bd65-1cf07926082c\" (UID: \"93000e3a-bf38-4568-bd65-1cf07926082c\") " Jul 28 00:25:58.185680 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185459 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-home" (OuterVolumeSpecName: "home") pod "93000e3a-bf38-4568-bd65-1cf07926082c" (UID: "93000e3a-bf38-4568-bd65-1cf07926082c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:58.185680 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185532 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93000e3a-bf38-4568-bd65-1cf07926082c" (UID: "93000e3a-bf38-4568-bd65-1cf07926082c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:58.185783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185688 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:58.185783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185707 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:58.185783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.185714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-model-cache" (OuterVolumeSpecName: "model-cache") pod "93000e3a-bf38-4568-bd65-1cf07926082c" (UID: "93000e3a-bf38-4568-bd65-1cf07926082c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:58.187465 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.187442 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93000e3a-bf38-4568-bd65-1cf07926082c-kube-api-access-frsnm" (OuterVolumeSpecName: "kube-api-access-frsnm") pod "93000e3a-bf38-4568-bd65-1cf07926082c" (UID: "93000e3a-bf38-4568-bd65-1cf07926082c"). InnerVolumeSpecName "kube-api-access-frsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:25:58.187551 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.187468 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93000e3a-bf38-4568-bd65-1cf07926082c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93000e3a-bf38-4568-bd65-1cf07926082c" (UID: "93000e3a-bf38-4568-bd65-1cf07926082c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:25:58.187551 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.187519 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-dshm" (OuterVolumeSpecName: "dshm") pod "93000e3a-bf38-4568-bd65-1cf07926082c" (UID: "93000e3a-bf38-4568-bd65-1cf07926082c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:58.286359 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.286313 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93000e3a-bf38-4568-bd65-1cf07926082c-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:58.286359 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.286355 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frsnm\" (UniqueName: \"kubernetes.io/projected/93000e3a-bf38-4568-bd65-1cf07926082c-kube-api-access-frsnm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:58.286359 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.286364 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:58.286359 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.286374 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93000e3a-bf38-4568-bd65-1cf07926082c-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:58.650009 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.649975 2555 generic.go:356] "Generic (PLEG): container finished" podID="93000e3a-bf38-4568-bd65-1cf07926082c" containerID="71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e" exitCode=0 Jul 28 00:25:58.650009 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.650018 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" Jul 28 00:25:58.650285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.650031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" event={"ID":"93000e3a-bf38-4568-bd65-1cf07926082c","Type":"ContainerDied","Data":"71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e"} Jul 28 00:25:58.650285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.650071 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff" event={"ID":"93000e3a-bf38-4568-bd65-1cf07926082c","Type":"ContainerDied","Data":"106bb1e2c344efe8f8cf80d3ab9113d60ad6587094d2b10938c95221d2cfd37b"} Jul 28 00:25:58.650285 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.650088 2555 scope.go:117] "RemoveContainer" containerID="71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e" Jul 28 00:25:58.657717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.657685 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" podUID="e7566239-a78e-4f7b-91f4-de5aa5bdca75" containerName="main" containerID="cri-o://be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3" gracePeriod=45 Jul 28 00:25:58.659633 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.659613 2555 scope.go:117] "RemoveContainer" containerID="71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e" Jul 28 00:25:58.659919 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:25:58.659891 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e\": container with ID starting with 71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e not found: ID does not exist" containerID="71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e" Jul 28 00:25:58.660011 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.659931 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e"} err="failed to get container status \"71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e\": rpc error: code = NotFound desc = could not find container \"71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e\": container with ID starting with 71ed7b02e9f705c2149380cf50c3a7180ac1d4151d768c3d076b69272e7e426e not found: ID does not exist" Jul 28 00:25:58.665337 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.665313 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff"] Jul 28 00:25:58.669922 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.669897 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-328ef1/rate-test-model-328ef1-kserve-565495b567-chcff"] Jul 28 00:25:58.918112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:58.918088 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:25:59.094633 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094540 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tmp-dir\") pod \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " Jul 28 00:25:59.094633 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094594 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-home\") pod \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " Jul 28 00:25:59.094868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094671 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-dshm\") pod \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " Jul 28 00:25:59.094868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094698 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-model-cache\") pod \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " Jul 28 00:25:59.094868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094738 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tls-certs\") pod \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " Jul 28 00:25:59.094868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094767 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97gpj\" (UniqueName: \"kubernetes.io/projected/e7566239-a78e-4f7b-91f4-de5aa5bdca75-kube-api-access-97gpj\") pod \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\" (UID: \"e7566239-a78e-4f7b-91f4-de5aa5bdca75\") " Jul 28 00:25:59.094868 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094835 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7566239-a78e-4f7b-91f4-de5aa5bdca75" (UID: "e7566239-a78e-4f7b-91f4-de5aa5bdca75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:59.095126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094844 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-home" (OuterVolumeSpecName: "home") pod "e7566239-a78e-4f7b-91f4-de5aa5bdca75" (UID: "e7566239-a78e-4f7b-91f4-de5aa5bdca75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:59.095126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.094984 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7566239-a78e-4f7b-91f4-de5aa5bdca75" (UID: "e7566239-a78e-4f7b-91f4-de5aa5bdca75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:59.095126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.095059 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:59.095126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.095069 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:59.095126 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.095078 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:59.096916 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.096885 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-dshm" (OuterVolumeSpecName: "dshm") pod "e7566239-a78e-4f7b-91f4-de5aa5bdca75" (UID: "e7566239-a78e-4f7b-91f4-de5aa5bdca75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:25:59.097024 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.096945 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7566239-a78e-4f7b-91f4-de5aa5bdca75" (UID: "e7566239-a78e-4f7b-91f4-de5aa5bdca75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:25:59.097024 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.096972 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7566239-a78e-4f7b-91f4-de5aa5bdca75-kube-api-access-97gpj" (OuterVolumeSpecName: "kube-api-access-97gpj") pod "e7566239-a78e-4f7b-91f4-de5aa5bdca75" (UID: "e7566239-a78e-4f7b-91f4-de5aa5bdca75"). InnerVolumeSpecName "kube-api-access-97gpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:25:59.195807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.195779 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7566239-a78e-4f7b-91f4-de5aa5bdca75-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:59.195807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.195805 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7566239-a78e-4f7b-91f4-de5aa5bdca75-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:59.195981 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.195817 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97gpj\" (UniqueName: \"kubernetes.io/projected/e7566239-a78e-4f7b-91f4-de5aa5bdca75-kube-api-access-97gpj\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:25:59.655619 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.655583 2555 generic.go:356] "Generic (PLEG): container finished" podID="e7566239-a78e-4f7b-91f4-de5aa5bdca75" containerID="be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3" exitCode=0 Jul 28 00:25:59.655798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.655633 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" Jul 28 00:25:59.655798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.655668 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" event={"ID":"e7566239-a78e-4f7b-91f4-de5aa5bdca75","Type":"ContainerDied","Data":"be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3"} Jul 28 00:25:59.655798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.655709 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp" event={"ID":"e7566239-a78e-4f7b-91f4-de5aa5bdca75","Type":"ContainerDied","Data":"be3072f57eacae365acc39a377e252534e86bdbc8de470979004fbf2e31527f8"} Jul 28 00:25:59.655798 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.655728 2555 scope.go:117] "RemoveContainer" containerID="be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3" Jul 28 00:25:59.664755 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.664739 2555 scope.go:117] "RemoveContainer" containerID="be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3" Jul 28 00:25:59.665013 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:25:59.664992 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3\": container with ID starting with be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3 not found: ID does not exist" containerID="be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3" Jul 28 00:25:59.665083 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.665020 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3"} err="failed to get container status \"be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3\": rpc error: code = NotFound desc = could not find container \"be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3\": container with ID starting with be045d8ab31ac71eec7ed370d58a481c024cffd0e931725038f0b235352bdad3 not found: ID does not exist" Jul 28 00:25:59.675627 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.675605 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp"] Jul 28 00:25:59.678841 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:25:59.678821 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6e175a/rate-test-model-6e175a-kserve-689897b4f9-scmjp"] Jul 28 00:26:00.565213 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:00.565181 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93000e3a-bf38-4568-bd65-1cf07926082c" path="/var/lib/kubelet/pods/93000e3a-bf38-4568-bd65-1cf07926082c/volumes" Jul 28 00:26:00.565599 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:00.565573 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7566239-a78e-4f7b-91f4-de5aa5bdca75" path="/var/lib/kubelet/pods/e7566239-a78e-4f7b-91f4-de5aa5bdca75/volumes" Jul 28 00:26:02.736839 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:02.736750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:03.055283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:03.055254 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:03.530964 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:03.530927 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:06.426214 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.426181 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:06.519694 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.519655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:06.788999 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.788963 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp"] Jul 28 00:26:06.789389 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.789376 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7566239-a78e-4f7b-91f4-de5aa5bdca75" containerName="main" Jul 28 00:26:06.789446 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.789390 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7566239-a78e-4f7b-91f4-de5aa5bdca75" containerName="main" Jul 28 00:26:06.789446 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.789407 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93000e3a-bf38-4568-bd65-1cf07926082c" containerName="main" Jul 28 00:26:06.789446 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.789412 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93000e3a-bf38-4568-bd65-1cf07926082c" containerName="main" Jul 28 00:26:06.789573 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.789486 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7566239-a78e-4f7b-91f4-de5aa5bdca75" containerName="main" Jul 28 00:26:06.789573 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.789508 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93000e3a-bf38-4568-bd65-1cf07926082c" containerName="main" Jul 28 00:26:06.794168 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.794141 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.796304 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.796275 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-74721a-serving-cert\"" Jul 28 00:26:06.807255 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.807178 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp"] Jul 28 00:26:06.823073 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.823039 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:06.865959 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.865925 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7db\" (UniqueName: \"kubernetes.io/projected/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-kube-api-access-dm7db\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.866122 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.865971 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.866122 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.865989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.966982 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.966949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7db\" (UniqueName: \"kubernetes.io/projected/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-kube-api-access-dm7db\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.967113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.966994 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.967113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.967015 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.967638 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.967612 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.969654 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.969631 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:06.973222 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.973183 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:06.974747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:06.974727 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7db\" (UniqueName: \"kubernetes.io/projected/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-kube-api-access-dm7db\") pod \"maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:07.105451 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.105365 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:07.248305 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.248133 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp"] Jul 28 00:26:07.251316 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:07.251284 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5d6af6_0adf_40c7_99bf_e7450d933c9c.slice/crio-a89e735b367dea292be4b658d3146a594fac7a1eaee07c01d51446dd798b10b7 WatchSource:0}: Error finding container a89e735b367dea292be4b658d3146a594fac7a1eaee07c01d51446dd798b10b7: Status 404 returned error can't find the container with id a89e735b367dea292be4b658d3146a594fac7a1eaee07c01d51446dd798b10b7 Jul 28 00:26:07.252924 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.252900 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:26:07.264822 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.264802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:07.360814 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.360718 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:07.394785 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.394744 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h"] Jul 28 00:26:07.399506 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.399469 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.401413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.401393 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-74721a\"" Jul 28 00:26:07.401546 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.401396 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-74721a-dockercfg-kzlk7\"" Jul 28 00:26:07.407861 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.407836 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h"] Jul 28 00:26:07.437677 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.437644 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq"] Jul 28 00:26:07.441161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.441145 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.450474 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.450450 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq"] Jul 28 00:26:07.471993 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.471957 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m5t4\" (UniqueName: \"kubernetes.io/projected/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-kube-api-access-7m5t4\") pod \"payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.472193 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.472017 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4585d2-bc83-4e60-8060-2084e9108ea4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.472193 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.472129 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.472303 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.472204 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddjs\" (UniqueName: \"kubernetes.io/projected/bc4585d2-bc83-4e60-8060-2084e9108ea4-kube-api-access-gddjs\") pod \"payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.572901 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.572857 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddjs\" (UniqueName: \"kubernetes.io/projected/bc4585d2-bc83-4e60-8060-2084e9108ea4-kube-api-access-gddjs\") pod \"payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.573088 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.572921 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m5t4\" (UniqueName: \"kubernetes.io/projected/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-kube-api-access-7m5t4\") pod \"payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.573088 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.572999 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4585d2-bc83-4e60-8060-2084e9108ea4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.573088 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.573074 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.573808 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.573781 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4585d2-bc83-4e60-8060-2084e9108ea4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.573876 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.573855 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.580191 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.580167 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m5t4\" (UniqueName: \"kubernetes.io/projected/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-kube-api-access-7m5t4\") pod \"payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.580941 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.580912 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddjs\" (UniqueName: \"kubernetes.io/projected/bc4585d2-bc83-4e60-8060-2084e9108ea4-kube-api-access-gddjs\") pod \"payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.701133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.701045 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" event={"ID":"7b5d6af6-0adf-40c7-99bf-e7450d933c9c","Type":"ContainerStarted","Data":"a89e735b367dea292be4b658d3146a594fac7a1eaee07c01d51446dd798b10b7"} Jul 28 00:26:07.711567 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.711532 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:07.751684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.751649 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:07.861054 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.861011 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h"] Jul 28 00:26:07.863101 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:07.863063 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4585d2_bc83_4e60_8060_2084e9108ea4.slice/crio-2447eacdbe3f16d65951ba16056fb358526c59396850322f5583403c2399e233 WatchSource:0}: Error finding container 2447eacdbe3f16d65951ba16056fb358526c59396850322f5583403c2399e233: Status 404 returned error can't find the container with id 2447eacdbe3f16d65951ba16056fb358526c59396850322f5583403c2399e233 Jul 28 00:26:07.910783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:07.910758 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq"] Jul 28 00:26:07.912749 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:07.912720 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca95e1ca_c7fa_40f6_a20a_f8de2258c3b6.slice/crio-5638448fa005b8d301571c711216e592aed66a61832c563a262b2166f40efe8b WatchSource:0}: Error finding container 5638448fa005b8d301571c711216e592aed66a61832c563a262b2166f40efe8b: Status 404 returned error can't find the container with id 5638448fa005b8d301571c711216e592aed66a61832c563a262b2166f40efe8b Jul 28 00:26:08.708857 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.708815 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" event={"ID":"7b5d6af6-0adf-40c7-99bf-e7450d933c9c","Type":"ContainerStarted","Data":"71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451"} Jul 28 00:26:08.709311 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.708924 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:08.710635 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.710599 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" event={"ID":"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6","Type":"ContainerStarted","Data":"4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737"} Jul 28 00:26:08.710764 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.710643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" event={"ID":"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6","Type":"ContainerStarted","Data":"5638448fa005b8d301571c711216e592aed66a61832c563a262b2166f40efe8b"} Jul 28 00:26:08.710764 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.710674 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:08.712336 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.712309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" event={"ID":"bc4585d2-bc83-4e60-8060-2084e9108ea4","Type":"ContainerStarted","Data":"8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c"} Jul 28 00:26:08.712455 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.712355 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" event={"ID":"bc4585d2-bc83-4e60-8060-2084e9108ea4","Type":"ContainerStarted","Data":"2447eacdbe3f16d65951ba16056fb358526c59396850322f5583403c2399e233"} Jul 28 00:26:08.712455 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.712378 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:08.724821 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.724760 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" podStartSLOduration=1.835603258 podStartE2EDuration="2.724741243s" totalImagesPullingTime="889.137985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:08.72301287 +0000 UTC m=+2968.647221749" watchObservedRunningTime="2026-07-28 00:26:08.724741243 +0000 UTC m=+2968.648950122" Jul 28 00:26:08.737673 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.737622 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" podStartSLOduration=1.7376021000000001 podStartE2EDuration="1.7376021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:08.737007076 +0000 UTC m=+2968.661215954" watchObservedRunningTime="2026-07-28 00:26:08.7376021 +0000 UTC m=+2968.661810975" Jul 28 00:26:08.752415 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:08.752363 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" podStartSLOduration=1.75234833 podStartE2EDuration="1.75234833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:08.750762167 +0000 UTC m=+2968.674971044" watchObservedRunningTime="2026-07-28 00:26:08.75234833 +0000 UTC m=+2968.676557207" Jul 28 00:26:10.863724 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:10.863674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:14.723818 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:14.723783 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:26:19.720984 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:19.720953 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:26:19.721460 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:19.721025 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:26:21.805969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:21.805933 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:23.324015 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:23.323966 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:25.520274 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.520230 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:25.596685 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.596629 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:25.597909 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.597884 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m"] Jul 28 00:26:25.601512 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.601472 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.603813 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.603790 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-056d1a-serving-cert\"" Jul 28 00:26:25.614296 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.614267 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m"] Jul 28 00:26:25.666138 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.666107 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:25.740099 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.740063 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f764s\" (UniqueName: \"kubernetes.io/projected/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-kube-api-access-f764s\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.740265 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.740112 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.740265 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.740206 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.841186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.841098 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f764s\" (UniqueName: \"kubernetes.io/projected/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-kube-api-access-f764s\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.841186 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.841140 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.841376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.841203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.841939 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.841913 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.843578 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.843555 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.848378 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.848357 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f764s\" (UniqueName: \"kubernetes.io/projected/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-kube-api-access-f764s\") pod \"maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:25.863329 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.863275 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:25.915747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:25.915721 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:26.057631 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.057603 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m"] Jul 28 00:26:26.059111 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:26.059080 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd971cc_1ba2_4b09_8583_74e6d60bcc11.slice/crio-89a90100b57a09acfcaa60df8bff63367fdc518882bfa1da707dfb012c2ecea0 WatchSource:0}: Error finding container 89a90100b57a09acfcaa60df8bff63367fdc518882bfa1da707dfb012c2ecea0: Status 404 returned error can't find the container with id 89a90100b57a09acfcaa60df8bff63367fdc518882bfa1da707dfb012c2ecea0 Jul 28 00:26:26.091933 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.091862 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:26.185732 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.185702 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:26.193826 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.193794 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg"] Jul 28 00:26:26.198447 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.198425 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.200671 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.200648 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-056d1a-dockercfg-bqpct\"" Jul 28 00:26:26.200787 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.200648 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-056d1a\"" Jul 28 00:26:26.206020 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.205996 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg"] Jul 28 00:26:26.244715 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.244682 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbk5c\" (UniqueName: \"kubernetes.io/projected/df5ba36e-100d-4eca-8417-5000754a55ca-kube-api-access-jbk5c\") pod \"payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.244922 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.244733 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ba36e-100d-4eca-8417-5000754a55ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.245306 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.245281 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l"] Jul 28 00:26:26.248708 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.248685 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l"] Jul 28 00:26:26.248819 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.248764 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.345894 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.345817 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ba36e-100d-4eca-8417-5000754a55ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.346047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.345901 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.346047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.345924 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbk5c\" (UniqueName: \"kubernetes.io/projected/df5ba36e-100d-4eca-8417-5000754a55ca-kube-api-access-jbk5c\") pod \"payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.346047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.345956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqk8m\" (UniqueName: \"kubernetes.io/projected/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-kube-api-access-pqk8m\") pod \"payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.346570 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.346538 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ba36e-100d-4eca-8417-5000754a55ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.353453 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.353429 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbk5c\" (UniqueName: \"kubernetes.io/projected/df5ba36e-100d-4eca-8417-5000754a55ca-kube-api-access-jbk5c\") pod \"payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.446986 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.446958 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqk8m\" (UniqueName: \"kubernetes.io/projected/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-kube-api-access-pqk8m\") pod \"payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.447083 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.447046 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.447683 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.447665 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.454105 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.454082 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqk8m\" (UniqueName: \"kubernetes.io/projected/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-kube-api-access-pqk8m\") pod \"payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.511974 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.511942 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.559543 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.559483 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.673105 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.673074 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg"] Jul 28 00:26:26.673730 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:26.673701 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5ba36e_100d_4eca_8417_5000754a55ca.slice/crio-a1be33616c3fad148225bf4ac76385a07f066452ae7761923cc55832e50e928c WatchSource:0}: Error finding container a1be33616c3fad148225bf4ac76385a07f066452ae7761923cc55832e50e928c: Status 404 returned error can't find the container with id a1be33616c3fad148225bf4ac76385a07f066452ae7761923cc55832e50e928c Jul 28 00:26:26.723944 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.723918 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l"] Jul 28 00:26:26.726878 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:26.726849 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e06d7b1_abb2_4cbb_be02_4f09a33aa29f.slice/crio-3cc9429b51134fe93dc56e0a8daa03f734a1cdb8d53af8e14f41590b5c9b374d WatchSource:0}: Error finding container 3cc9429b51134fe93dc56e0a8daa03f734a1cdb8d53af8e14f41590b5c9b374d: Status 404 returned error can't find the container with id 3cc9429b51134fe93dc56e0a8daa03f734a1cdb8d53af8e14f41590b5c9b374d Jul 28 00:26:26.816474 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.816182 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" event={"ID":"df5ba36e-100d-4eca-8417-5000754a55ca","Type":"ContainerStarted","Data":"b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61"} Jul 28 00:26:26.816474 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.816235 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" event={"ID":"df5ba36e-100d-4eca-8417-5000754a55ca","Type":"ContainerStarted","Data":"a1be33616c3fad148225bf4ac76385a07f066452ae7761923cc55832e50e928c"} Jul 28 00:26:26.816474 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.816392 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:26.818134 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.818109 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" event={"ID":"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f","Type":"ContainerStarted","Data":"a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179"} Jul 28 00:26:26.818268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.818147 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" event={"ID":"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f","Type":"ContainerStarted","Data":"3cc9429b51134fe93dc56e0a8daa03f734a1cdb8d53af8e14f41590b5c9b374d"} Jul 28 00:26:26.818268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.818213 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:26.819833 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.819813 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" event={"ID":"5fd971cc-1ba2-4b09-8583-74e6d60bcc11","Type":"ContainerStarted","Data":"aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d"} Jul 28 00:26:26.819934 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.819856 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" event={"ID":"5fd971cc-1ba2-4b09-8583-74e6d60bcc11","Type":"ContainerStarted","Data":"89a90100b57a09acfcaa60df8bff63367fdc518882bfa1da707dfb012c2ecea0"} Jul 28 00:26:26.819934 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.819888 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:26.851663 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.851609 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" podStartSLOduration=0.851589926 podStartE2EDuration="851.589926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:26.848526609 +0000 UTC m=+2986.772735486" watchObservedRunningTime="2026-07-28 00:26:26.851589926 +0000 UTC m=+2986.775798806" Jul 28 00:26:26.879293 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.879208 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" podStartSLOduration=0.879192607 podStartE2EDuration="879.192607ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:26.874810883 +0000 UTC m=+2986.799019761" watchObservedRunningTime="2026-07-28 00:26:26.879192607 +0000 UTC m=+2986.803401485" Jul 28 00:26:26.905872 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:26.905826 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" podStartSLOduration=1.5420654950000001 podStartE2EDuration="1.905809565s" totalImagesPullingTime="363.74407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:26.904954711 +0000 UTC m=+2986.829163590" watchObservedRunningTime="2026-07-28 00:26:26.905809565 +0000 UTC m=+2986.830018442" Jul 28 00:26:30.053405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:30.053356 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:32.831548 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:32.831518 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:26:37.827896 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:37.827855 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:26:37.828389 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:37.827932 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:26:47.321891 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:47.321857 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tlfb6_cb094531-c0c3-4729-a052-0b2cd6e835d2/payload-processing/0.log" Jul 28 00:26:47.450581 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:47.450552 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l_5e06d7b1-abb2-4cbb-be02-4f09a33aa29f/payload-processing/0.log" Jul 28 00:26:56.243841 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.243796 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg"] Jul 28 00:26:56.248167 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.248146 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.250437 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.250410 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-74721a\"/\"openshift-service-ca.crt\"" Jul 28 00:26:56.250437 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.250428 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-74721a\"/\"kube-root-ca.crt\"" Jul 28 00:26:56.251442 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.251415 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-74721a\"/\"default-dockercfg-lfb5g\"" Jul 28 00:26:56.251580 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.251519 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-74721a\"/\"ipp-route-74721a-kserve-self-signed-certs\"" Jul 28 00:26:56.255182 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.255078 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg"] Jul 28 00:26:56.324060 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.324030 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:26:56.325136 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.325111 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tls-certs\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.325242 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.325227 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-dshm\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.325317 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.325256 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-model-cache\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.325362 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.325322 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tmp-dir\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.325362 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.325346 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9gq\" (UniqueName: \"kubernetes.io/projected/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-kube-api-access-mq9gq\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.325440 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.325383 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-home\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.426665 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.426627 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-dshm\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.426665 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.426665 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-model-cache\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.426927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.426710 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tmp-dir\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.426927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.426729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9gq\" (UniqueName: \"kubernetes.io/projected/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-kube-api-access-mq9gq\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.426927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.426753 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-home\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.426927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.426799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tls-certs\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.427208 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.427186 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-model-cache\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.427272 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.427239 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-home\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.427272 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.427254 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tmp-dir\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.428996 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.428975 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-dshm\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.429290 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.429274 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tls-certs\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.434956 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.434930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9gq\" (UniqueName: \"kubernetes.io/projected/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-kube-api-access-mq9gq\") pod \"ipp-route-74721a-kserve-556ff4f899-zjzdg\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.559989 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.559899 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:26:56.700516 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.700476 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg"] Jul 28 00:26:56.704577 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:26:56.704486 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod485fbc7e_6a4f_442f_bef3_1fa8be747ad5.slice/crio-b2e2d7f8657308c1977bea5a3de2b642ed0bac2b20bba35086705ff2f1520efc WatchSource:0}: Error finding container b2e2d7f8657308c1977bea5a3de2b642ed0bac2b20bba35086705ff2f1520efc: Status 404 returned error can't find the container with id b2e2d7f8657308c1977bea5a3de2b642ed0bac2b20bba35086705ff2f1520efc Jul 28 00:26:56.993364 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.993330 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" event={"ID":"485fbc7e-6a4f-442f-bef3-1fa8be747ad5","Type":"ContainerStarted","Data":"59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d"} Jul 28 00:26:56.993551 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:56.993376 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" event={"ID":"485fbc7e-6a4f-442f-bef3-1fa8be747ad5","Type":"ContainerStarted","Data":"b2e2d7f8657308c1977bea5a3de2b642ed0bac2b20bba35086705ff2f1520efc"} Jul 28 00:26:57.009356 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:26:57.009296 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" podStartSLOduration=1.009275467 podStartE2EDuration="1.009275467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:26:57.008237934 +0000 UTC m=+3016.932446812" watchObservedRunningTime="2026-07-28 00:26:57.009275467 +0000 UTC m=+3016.933484349" Jul 28 00:27:06.566302 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:06.566269 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:27:06.566835 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:06.566314 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:27:06.573283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:06.573258 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:27:07.062605 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:07.062577 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:27:44.693066 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:44.692982 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h_bc4585d2-bc83-4e60-8060-2084e9108ea4/payload-processing/0.log" Jul 28 00:27:44.826648 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:44.826624 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tlfb6_cb094531-c0c3-4729-a052-0b2cd6e835d2/payload-processing/0.log" Jul 28 00:27:44.960325 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:44.960239 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l_5e06d7b1-abb2-4cbb-be02-4f09a33aa29f/payload-processing/0.log" Jul 28 00:27:45.247672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:45.247640 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:45.625771 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:45.625664 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:45.975383 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:45.975286 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:51.557329 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:51.557290 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:51.637501 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:51.637471 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:53.364863 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:53.364805 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:53.882965 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:53.882918 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:53.968794 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:53.968760 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:27:56.058572 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:27:56.058542 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:04.592673 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:04.592620 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:04.693310 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:04.693269 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:12.134593 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.134558 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz"] Jul 28 00:28:12.138190 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.138171 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:12.140114 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.140097 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:28:12.143983 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.143955 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz"] Jul 28 00:28:12.232564 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.232534 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/050cfd80-f216-49d0-bf09-6942e212382a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz\" (UID: \"050cfd80-f216-49d0-bf09-6942e212382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:12.333949 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.333894 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/050cfd80-f216-49d0-bf09-6942e212382a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz\" (UID: \"050cfd80-f216-49d0-bf09-6942e212382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:12.334804 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.334776 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/050cfd80-f216-49d0-bf09-6942e212382a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz\" (UID: \"050cfd80-f216-49d0-bf09-6942e212382a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:12.450251 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.450165 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:12.587067 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:12.587040 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz"] Jul 28 00:28:12.589827 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:28:12.589796 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050cfd80_f216_49d0_bf09_6942e212382a.slice/crio-8c20b50e350b5aa2ef9d12397e64077a81b12ebc6c45e06234c2dee1ea25930f WatchSource:0}: Error finding container 8c20b50e350b5aa2ef9d12397e64077a81b12ebc6c45e06234c2dee1ea25930f: Status 404 returned error can't find the container with id 8c20b50e350b5aa2ef9d12397e64077a81b12ebc6c45e06234c2dee1ea25930f Jul 28 00:28:13.402422 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:13.402384 2555 generic.go:356] "Generic (PLEG): container finished" podID="050cfd80-f216-49d0-bf09-6942e212382a" containerID="04037bddf1397d3bc8ec048be9c2bf6debd3c4d991729736a5a24d4978c9ad17" exitCode=0 Jul 28 00:28:13.402819 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:13.402474 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" event={"ID":"050cfd80-f216-49d0-bf09-6942e212382a","Type":"ContainerDied","Data":"04037bddf1397d3bc8ec048be9c2bf6debd3c4d991729736a5a24d4978c9ad17"} Jul 28 00:28:13.402819 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:13.402536 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" event={"ID":"050cfd80-f216-49d0-bf09-6942e212382a","Type":"ContainerStarted","Data":"8c20b50e350b5aa2ef9d12397e64077a81b12ebc6c45e06234c2dee1ea25930f"} Jul 28 00:28:14.553825 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:14.553799 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:14.657054 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:14.657016 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/050cfd80-f216-49d0-bf09-6942e212382a-maas-api-service-ca\") pod \"050cfd80-f216-49d0-bf09-6942e212382a\" (UID: \"050cfd80-f216-49d0-bf09-6942e212382a\") " Jul 28 00:28:14.657392 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:14.657355 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050cfd80-f216-49d0-bf09-6942e212382a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "050cfd80-f216-49d0-bf09-6942e212382a" (UID: "050cfd80-f216-49d0-bf09-6942e212382a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:28:14.758385 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:14.758356 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/050cfd80-f216-49d0-bf09-6942e212382a-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:15.414420 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:15.414385 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" Jul 28 00:28:15.414681 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:15.414381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz" event={"ID":"050cfd80-f216-49d0-bf09-6942e212382a","Type":"ContainerDied","Data":"8c20b50e350b5aa2ef9d12397e64077a81b12ebc6c45e06234c2dee1ea25930f"} Jul 28 00:28:15.414681 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:15.414567 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c20b50e350b5aa2ef9d12397e64077a81b12ebc6c45e06234c2dee1ea25930f" Jul 28 00:28:23.145216 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:23.145183 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:23.234217 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:23.234182 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:23.296970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:23.296940 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:27.201966 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:27.201929 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz"] Jul 28 00:28:27.204478 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:27.204436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:27.207564 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:27.207529 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d194aa-4dcqz"] Jul 28 00:28:27.581344 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:27.581311 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:28.566555 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:28.566525 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050cfd80-f216-49d0-bf09-6942e212382a" path="/var/lib/kubelet/pods/050cfd80-f216-49d0-bf09-6942e212382a/volumes" Jul 28 00:28:33.633133 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:33.633098 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:33.749353 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:33.749320 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:33.933552 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:33.933456 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:34.163401 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:34.163369 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:54.569997 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.569961 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp"] Jul 28 00:28:54.570607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.570257 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" podUID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" containerName="maas-api" containerID="cri-o://71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451" gracePeriod=30 Jul 28 00:28:54.589807 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.589781 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:54.612108 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.612068 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h"] Jul 28 00:28:54.612329 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.612302 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" podUID="bc4585d2-bc83-4e60-8060-2084e9108ea4" containerName="payload-processing" containerID="cri-o://8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c" gracePeriod=30 Jul 28 00:28:54.616900 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.616868 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq"] Jul 28 00:28:54.617190 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.617086 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" podUID="ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" containerName="payload-pre-processing" containerID="cri-o://4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737" gracePeriod=30 Jul 28 00:28:54.690532 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.690473 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:54.719725 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.719689 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" podUID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.104:8443/health\": dial tcp 10.133.0.104:8443: connect: connection refused" Jul 28 00:28:54.791171 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.790934 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:54.885407 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.885371 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:28:54.953856 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.953826 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:28:54.956765 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:54.956745 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:28:55.025183 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025148 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm7db\" (UniqueName: \"kubernetes.io/projected/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-kube-api-access-dm7db\") pod \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " Jul 28 00:28:55.025360 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025200 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-openshift-service-ca\") pod \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " Jul 28 00:28:55.025360 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025243 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m5t4\" (UniqueName: \"kubernetes.io/projected/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-kube-api-access-7m5t4\") pod \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " Jul 28 00:28:55.025360 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025270 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-plugins-config-volume\") pod \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\" (UID: \"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6\") " Jul 28 00:28:55.025360 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025308 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-maas-api-tls\") pod \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\" (UID: \"7b5d6af6-0adf-40c7-99bf-e7450d933c9c\") " Jul 28 00:28:55.025360 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025331 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4585d2-bc83-4e60-8060-2084e9108ea4-plugins-config-volume\") pod \"bc4585d2-bc83-4e60-8060-2084e9108ea4\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " Jul 28 00:28:55.025658 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025387 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gddjs\" (UniqueName: \"kubernetes.io/projected/bc4585d2-bc83-4e60-8060-2084e9108ea4-kube-api-access-gddjs\") pod \"bc4585d2-bc83-4e60-8060-2084e9108ea4\" (UID: \"bc4585d2-bc83-4e60-8060-2084e9108ea4\") " Jul 28 00:28:55.025713 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025683 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b5d6af6-0adf-40c7-99bf-e7450d933c9c" (UID: "7b5d6af6-0adf-40c7-99bf-e7450d933c9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:28:55.025767 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025704 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" (UID: "ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:28:55.025862 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.025834 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4585d2-bc83-4e60-8060-2084e9108ea4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc4585d2-bc83-4e60-8060-2084e9108ea4" (UID: "bc4585d2-bc83-4e60-8060-2084e9108ea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:28:55.027472 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.027443 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-kube-api-access-dm7db" (OuterVolumeSpecName: "kube-api-access-dm7db") pod "7b5d6af6-0adf-40c7-99bf-e7450d933c9c" (UID: "7b5d6af6-0adf-40c7-99bf-e7450d933c9c"). InnerVolumeSpecName "kube-api-access-dm7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:28:55.027472 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.027459 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-kube-api-access-7m5t4" (OuterVolumeSpecName: "kube-api-access-7m5t4") pod "ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" (UID: "ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6"). InnerVolumeSpecName "kube-api-access-7m5t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:28:55.027472 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.027458 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b5d6af6-0adf-40c7-99bf-e7450d933c9c" (UID: "7b5d6af6-0adf-40c7-99bf-e7450d933c9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:28:55.027691 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.027516 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4585d2-bc83-4e60-8060-2084e9108ea4-kube-api-access-gddjs" (OuterVolumeSpecName: "kube-api-access-gddjs") pod "bc4585d2-bc83-4e60-8060-2084e9108ea4" (UID: "bc4585d2-bc83-4e60-8060-2084e9108ea4"). InnerVolumeSpecName "kube-api-access-gddjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127048 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m5t4\" (UniqueName: \"kubernetes.io/projected/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-kube-api-access-7m5t4\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127079 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127092 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127104 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4585d2-bc83-4e60-8060-2084e9108ea4-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127115 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddjs\" (UniqueName: \"kubernetes.io/projected/bc4585d2-bc83-4e60-8060-2084e9108ea4-kube-api-access-gddjs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127127 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7db\" (UniqueName: \"kubernetes.io/projected/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-kube-api-access-dm7db\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.127140 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.127140 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5d6af6-0adf-40c7-99bf-e7450d933c9c-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:28:55.638954 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.638918 2555 generic.go:356] "Generic (PLEG): container finished" podID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" containerID="71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451" exitCode=0 Jul 28 00:28:55.639376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.638968 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" Jul 28 00:28:55.639376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.639001 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" event={"ID":"7b5d6af6-0adf-40c7-99bf-e7450d933c9c","Type":"ContainerDied","Data":"71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451"} Jul 28 00:28:55.639376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.639058 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp" event={"ID":"7b5d6af6-0adf-40c7-99bf-e7450d933c9c","Type":"ContainerDied","Data":"a89e735b367dea292be4b658d3146a594fac7a1eaee07c01d51446dd798b10b7"} Jul 28 00:28:55.639376 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.639092 2555 scope.go:117] "RemoveContainer" containerID="71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451" Jul 28 00:28:55.640389 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.640334 2555 generic.go:356] "Generic (PLEG): container finished" podID="ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" containerID="4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737" exitCode=0 Jul 28 00:28:55.640472 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.640387 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" Jul 28 00:28:55.640472 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.640422 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" event={"ID":"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6","Type":"ContainerDied","Data":"4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737"} Jul 28 00:28:55.640607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.640485 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq" event={"ID":"ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6","Type":"ContainerDied","Data":"5638448fa005b8d301571c711216e592aed66a61832c563a262b2166f40efe8b"} Jul 28 00:28:55.641916 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.641884 2555 generic.go:356] "Generic (PLEG): container finished" podID="bc4585d2-bc83-4e60-8060-2084e9108ea4" containerID="8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c" exitCode=0 Jul 28 00:28:55.642025 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.641919 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" event={"ID":"bc4585d2-bc83-4e60-8060-2084e9108ea4","Type":"ContainerDied","Data":"8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c"} Jul 28 00:28:55.642025 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.641935 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" Jul 28 00:28:55.642025 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.641961 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h" event={"ID":"bc4585d2-bc83-4e60-8060-2084e9108ea4","Type":"ContainerDied","Data":"2447eacdbe3f16d65951ba16056fb358526c59396850322f5583403c2399e233"} Jul 28 00:28:55.649416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.649218 2555 scope.go:117] "RemoveContainer" containerID="71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451" Jul 28 00:28:55.649708 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:28:55.649677 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451\": container with ID starting with 71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451 not found: ID does not exist" containerID="71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451" Jul 28 00:28:55.649825 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.649718 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451"} err="failed to get container status \"71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451\": rpc error: code = NotFound desc = could not find container \"71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451\": container with ID starting with 71cd47c5a426efce56e1b2c7ec8ba31bc87c6a6f58058a1266b058f5d7ba5451 not found: ID does not exist" Jul 28 00:28:55.649825 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.649743 2555 scope.go:117] "RemoveContainer" containerID="4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737" Jul 28 00:28:55.659069 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.659049 2555 scope.go:117] "RemoveContainer" containerID="4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737" Jul 28 00:28:55.659703 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:28:55.659676 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737\": container with ID starting with 4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737 not found: ID does not exist" containerID="4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737" Jul 28 00:28:55.659822 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.659711 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737"} err="failed to get container status \"4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737\": rpc error: code = NotFound desc = could not find container \"4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737\": container with ID starting with 4f82f44cd6f7f0308a5a6e5f2ef53060972b509d215dbf6a17d6aea54ccec737 not found: ID does not exist" Jul 28 00:28:55.659822 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.659736 2555 scope.go:117] "RemoveContainer" containerID="8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c" Jul 28 00:28:55.661903 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.661887 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp"] Jul 28 00:28:55.665878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.665860 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-74721a-5f86df5cff-r7nzp"] Jul 28 00:28:55.668625 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.668612 2555 scope.go:117] "RemoveContainer" containerID="8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c" Jul 28 00:28:55.668877 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:28:55.668861 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c\": container with ID starting with 8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c not found: ID does not exist" containerID="8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c" Jul 28 00:28:55.668917 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.668884 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c"} err="failed to get container status \"8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c\": rpc error: code = NotFound desc = could not find container \"8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c\": container with ID starting with 8107b0bda531bc851f69845bfec219caec5a7668b0f5d566b24becf98b13de9c not found: ID does not exist" Jul 28 00:28:55.674456 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.674436 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h"] Jul 28 00:28:55.680691 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.680670 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-74721a-6fc8b79d78-qxx8h"] Jul 28 00:28:55.690170 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.690147 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq"] Jul 28 00:28:55.695603 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:55.695569 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-74721a-6bcf96fdbd-tzjnq"] Jul 28 00:28:56.572252 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:56.572215 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" path="/var/lib/kubelet/pods/7b5d6af6-0adf-40c7-99bf-e7450d933c9c/volumes" Jul 28 00:28:56.572658 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:56.572642 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc4585d2-bc83-4e60-8060-2084e9108ea4" path="/var/lib/kubelet/pods/bc4585d2-bc83-4e60-8060-2084e9108ea4/volumes" Jul 28 00:28:56.572979 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:56.572967 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" path="/var/lib/kubelet/pods/ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6/volumes" Jul 28 00:28:57.785920 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:57.785883 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:28:58.186907 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:28:58.186819 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:29:03.168816 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:03.168724 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg"] Jul 28 00:29:03.261097 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:03.261065 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:29:18.188253 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.188177 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" podUID="485fbc7e-6a4f-442f-bef3-1fa8be747ad5" containerName="main" containerID="cri-o://59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d" gracePeriod=45 Jul 28 00:29:18.450892 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.450838 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:29:18.539291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539263 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-dshm\") pod \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " Jul 28 00:29:18.539291 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539295 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tmp-dir\") pod \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " Jul 28 00:29:18.539562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539346 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-home\") pod \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " Jul 28 00:29:18.539562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539395 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq9gq\" (UniqueName: \"kubernetes.io/projected/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-kube-api-access-mq9gq\") pod \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " Jul 28 00:29:18.539562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539449 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tls-certs\") pod \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " Jul 28 00:29:18.539562 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539477 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-model-cache\") pod \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\" (UID: \"485fbc7e-6a4f-442f-bef3-1fa8be747ad5\") " Jul 28 00:29:18.539780 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539617 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-home" (OuterVolumeSpecName: "home") pod "485fbc7e-6a4f-442f-bef3-1fa8be747ad5" (UID: "485fbc7e-6a4f-442f-bef3-1fa8be747ad5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:29:18.539780 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539673 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "485fbc7e-6a4f-442f-bef3-1fa8be747ad5" (UID: "485fbc7e-6a4f-442f-bef3-1fa8be747ad5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:29:18.539876 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539802 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-model-cache" (OuterVolumeSpecName: "model-cache") pod "485fbc7e-6a4f-442f-bef3-1fa8be747ad5" (UID: "485fbc7e-6a4f-442f-bef3-1fa8be747ad5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:29:18.539921 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539885 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-home\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:29:18.539921 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539904 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-model-cache\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:29:18.539921 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.539919 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tmp-dir\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:29:18.541598 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.541565 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-dshm" (OuterVolumeSpecName: "dshm") pod "485fbc7e-6a4f-442f-bef3-1fa8be747ad5" (UID: "485fbc7e-6a4f-442f-bef3-1fa8be747ad5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:29:18.541714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.541634 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "485fbc7e-6a4f-442f-bef3-1fa8be747ad5" (UID: "485fbc7e-6a4f-442f-bef3-1fa8be747ad5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:29:18.541714 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.541650 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-kube-api-access-mq9gq" (OuterVolumeSpecName: "kube-api-access-mq9gq") pod "485fbc7e-6a4f-442f-bef3-1fa8be747ad5" (UID: "485fbc7e-6a4f-442f-bef3-1fa8be747ad5"). InnerVolumeSpecName "kube-api-access-mq9gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:29:18.640676 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.640648 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9gq\" (UniqueName: \"kubernetes.io/projected/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-kube-api-access-mq9gq\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:29:18.640676 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.640673 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-tls-certs\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:29:18.640676 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.640685 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/485fbc7e-6a4f-442f-bef3-1fa8be747ad5-dshm\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:29:18.762681 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.762645 2555 generic.go:356] "Generic (PLEG): container finished" podID="485fbc7e-6a4f-442f-bef3-1fa8be747ad5" containerID="59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d" exitCode=0 Jul 28 00:29:18.762846 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.762695 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" Jul 28 00:29:18.762846 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.762740 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" event={"ID":"485fbc7e-6a4f-442f-bef3-1fa8be747ad5","Type":"ContainerDied","Data":"59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d"} Jul 28 00:29:18.762846 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.762790 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg" event={"ID":"485fbc7e-6a4f-442f-bef3-1fa8be747ad5","Type":"ContainerDied","Data":"b2e2d7f8657308c1977bea5a3de2b642ed0bac2b20bba35086705ff2f1520efc"} Jul 28 00:29:18.762846 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.762811 2555 scope.go:117] "RemoveContainer" containerID="59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d" Jul 28 00:29:18.771404 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.771373 2555 scope.go:117] "RemoveContainer" containerID="59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d" Jul 28 00:29:18.771690 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:29:18.771665 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d\": container with ID starting with 59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d not found: ID does not exist" containerID="59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d" Jul 28 00:29:18.771760 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.771703 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d"} err="failed to get container status \"59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d\": rpc error: code = NotFound desc = could not find container \"59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d\": container with ID starting with 59f7e1402ad7df8b309d500165a1878ec8952442b91a9daffcb5f9882338649d not found: ID does not exist" Jul 28 00:29:18.777437 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.777415 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg"] Jul 28 00:29:18.780936 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.780917 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-74721a/ipp-route-74721a-kserve-556ff4f899-zjzdg"] Jul 28 00:29:18.845648 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:18.845619 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:29:20.567047 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:20.567014 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485fbc7e-6a4f-442f-bef3-1fa8be747ad5" path="/var/lib/kubelet/pods/485fbc7e-6a4f-442f-bef3-1fa8be747ad5/volumes" Jul 28 00:29:40.271184 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:40.271130 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:29:40.423105 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:40.423071 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:29:46.076835 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:29:46.076799 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:00.139682 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.139645 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5"] Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140071 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485fbc7e-6a4f-442f-bef3-1fa8be747ad5" containerName="main" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140083 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="485fbc7e-6a4f-442f-bef3-1fa8be747ad5" containerName="main" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140099 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050cfd80-f216-49d0-bf09-6942e212382a" containerName="revoke-keys" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140105 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="050cfd80-f216-49d0-bf09-6942e212382a" containerName="revoke-keys" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140118 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4585d2-bc83-4e60-8060-2084e9108ea4" containerName="payload-processing" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140124 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4585d2-bc83-4e60-8060-2084e9108ea4" containerName="payload-processing" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140132 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" containerName="maas-api" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140137 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" containerName="maas-api" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140148 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" containerName="payload-pre-processing" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140155 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" containerName="payload-pre-processing" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140250 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca95e1ca-c7fa-40f6-a20a-f8de2258c3b6" containerName="payload-pre-processing" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140265 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="050cfd80-f216-49d0-bf09-6942e212382a" containerName="revoke-keys" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140276 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5d6af6-0adf-40c7-99bf-e7450d933c9c" containerName="maas-api" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140285 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="485fbc7e-6a4f-442f-bef3-1fa8be747ad5" containerName="main" Jul 28 00:30:00.142142 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.140296 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4585d2-bc83-4e60-8060-2084e9108ea4" containerName="payload-processing" Jul 28 00:30:00.143530 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.143509 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r"] Jul 28 00:30:00.143638 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.143616 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" Jul 28 00:30:00.145792 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.145766 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:30:00.147110 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.147090 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2"] Jul 28 00:30:00.147253 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.147224 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" Jul 28 00:30:00.156585 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.155888 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg"] Jul 28 00:30:00.157165 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.157143 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" Jul 28 00:30:00.160037 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.160014 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" Jul 28 00:30:00.160230 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.160204 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" Jul 28 00:30:00.160864 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.160130 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5"] Jul 28 00:30:00.160864 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.160545 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r"] Jul 28 00:30:00.161990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.161968 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2"] Jul 28 00:30:00.166666 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.166530 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg"] Jul 28 00:30:00.171051 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.171030 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" Jul 28 00:30:00.219315 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.219286 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" Jul 28 00:30:00.227292 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.227262 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" Jul 28 00:30:00.406176 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.405791 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r"] Jul 28 00:30:00.407457 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:00.407433 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a1137c_1853_4d80_a4d8_e4b63361fbda.slice/crio-5f281fe1c38f71aab8809bdaca5f6840ea5533ebab1c8a4b601b8faf72cfa289 WatchSource:0}: Error finding container 5f281fe1c38f71aab8809bdaca5f6840ea5533ebab1c8a4b601b8faf72cfa289: Status 404 returned error can't find the container with id 5f281fe1c38f71aab8809bdaca5f6840ea5533ebab1c8a4b601b8faf72cfa289 Jul 28 00:30:00.442078 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.442041 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2"] Jul 28 00:30:00.447873 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:00.447845 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983cabd4_fcab_4b6c_a2b0_d10b966c0b29.slice/crio-428bd1ed689b34ab91b2dc95af7d0711c90126c11e3ae219ebfc62f89231abd9 WatchSource:0}: Error finding container 428bd1ed689b34ab91b2dc95af7d0711c90126c11e3ae219ebfc62f89231abd9: Status 404 returned error can't find the container with id 428bd1ed689b34ab91b2dc95af7d0711c90126c11e3ae219ebfc62f89231abd9 Jul 28 00:30:00.471607 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.471525 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg"] Jul 28 00:30:00.473648 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:00.473624 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6142d31d_0d77_4b36_be26_8aa93752abc9.slice/crio-10e75c22e27cf3a6a7ce313d868d3c55330d10dbd37453667026e3da1d0564fc WatchSource:0}: Error finding container 10e75c22e27cf3a6a7ce313d868d3c55330d10dbd37453667026e3da1d0564fc: Status 404 returned error can't find the container with id 10e75c22e27cf3a6a7ce313d868d3c55330d10dbd37453667026e3da1d0564fc Jul 28 00:30:00.570468 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.570444 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5"] Jul 28 00:30:00.572524 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:00.572484 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298f7354_fe28_4aeb_8a8d_21c389e24a78.slice/crio-b936de0e394d4a358be7734a62f6773ac225a747655ab6f10471df9895e8960a WatchSource:0}: Error finding container b936de0e394d4a358be7734a62f6773ac225a747655ab6f10471df9895e8960a: Status 404 returned error can't find the container with id b936de0e394d4a358be7734a62f6773ac225a747655ab6f10471df9895e8960a Jul 28 00:30:00.988413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.988290 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" event={"ID":"66a1137c-1853-4d80-a4d8-e4b63361fbda","Type":"ContainerStarted","Data":"8befa2c870e747e9bc3369c3a3d28bb5f2d37d1b203b2e82204e084b93f16819"} Jul 28 00:30:00.988413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.988364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" event={"ID":"66a1137c-1853-4d80-a4d8-e4b63361fbda","Type":"ContainerStarted","Data":"5f281fe1c38f71aab8809bdaca5f6840ea5533ebab1c8a4b601b8faf72cfa289"} Jul 28 00:30:00.989908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.989881 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" event={"ID":"983cabd4-fcab-4b6c-a2b0-d10b966c0b29","Type":"ContainerStarted","Data":"0ec1aca5d159ff2b36840be32c963eee912d577f011460d6f0c8930f4cd7df65"} Jul 28 00:30:00.990039 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.989932 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" event={"ID":"983cabd4-fcab-4b6c-a2b0-d10b966c0b29","Type":"ContainerStarted","Data":"428bd1ed689b34ab91b2dc95af7d0711c90126c11e3ae219ebfc62f89231abd9"} Jul 28 00:30:00.991555 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.991528 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" event={"ID":"6142d31d-0d77-4b36-be26-8aa93752abc9","Type":"ContainerStarted","Data":"712122ab3b7496af541edf078d9541717013f896da9cbfbe69f7c7eebe77f9a6"} Jul 28 00:30:00.991648 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.991583 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" event={"ID":"6142d31d-0d77-4b36-be26-8aa93752abc9","Type":"ContainerStarted","Data":"10e75c22e27cf3a6a7ce313d868d3c55330d10dbd37453667026e3da1d0564fc"} Jul 28 00:30:00.993077 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.993058 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" event={"ID":"298f7354-fe28-4aeb-8a8d-21c389e24a78","Type":"ContainerStarted","Data":"7b78b3f96eadf83499e70b331e518d80335d11ccd1dded6b617e778ec49686d4"} Jul 28 00:30:00.993154 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:00.993092 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" event={"ID":"298f7354-fe28-4aeb-8a8d-21c389e24a78","Type":"ContainerStarted","Data":"b936de0e394d4a358be7734a62f6773ac225a747655ab6f10471df9895e8960a"} Jul 28 00:30:01.001388 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:01.001341 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" podStartSLOduration=1.001327898 podStartE2EDuration="1.001327898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:30:01.000909984 +0000 UTC m=+3200.925118863" watchObservedRunningTime="2026-07-28 00:30:01.001327898 +0000 UTC m=+3200.925536794" Jul 28 00:30:01.014809 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:01.014751 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" podStartSLOduration=1.014739238 podStartE2EDuration="1.014739238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:30:01.013873818 +0000 UTC m=+3200.938082707" watchObservedRunningTime="2026-07-28 00:30:01.014739238 +0000 UTC m=+3200.938948116" Jul 28 00:30:01.028905 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:01.028852 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" podStartSLOduration=1.028837777 podStartE2EDuration="1.028837777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:30:01.025806906 +0000 UTC m=+3200.950015784" watchObservedRunningTime="2026-07-28 00:30:01.028837777 +0000 UTC m=+3200.953046655" Jul 28 00:30:01.039758 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:01.039715 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" podStartSLOduration=1.039703085 podStartE2EDuration="1.039703085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:30:01.038109741 +0000 UTC m=+3200.962318620" watchObservedRunningTime="2026-07-28 00:30:01.039703085 +0000 UTC m=+3200.963911962" Jul 28 00:30:01.999620 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:01.999582 2555 generic.go:356] "Generic (PLEG): container finished" podID="298f7354-fe28-4aeb-8a8d-21c389e24a78" containerID="7b78b3f96eadf83499e70b331e518d80335d11ccd1dded6b617e778ec49686d4" exitCode=0 Jul 28 00:30:02.000083 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:01.999674 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" event={"ID":"298f7354-fe28-4aeb-8a8d-21c389e24a78","Type":"ContainerDied","Data":"7b78b3f96eadf83499e70b331e518d80335d11ccd1dded6b617e778ec49686d4"} Jul 28 00:30:02.001303 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:02.001281 2555 generic.go:356] "Generic (PLEG): container finished" podID="66a1137c-1853-4d80-a4d8-e4b63361fbda" containerID="8befa2c870e747e9bc3369c3a3d28bb5f2d37d1b203b2e82204e084b93f16819" exitCode=0 Jul 28 00:30:02.001415 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:02.001308 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" event={"ID":"66a1137c-1853-4d80-a4d8-e4b63361fbda","Type":"ContainerDied","Data":"8befa2c870e747e9bc3369c3a3d28bb5f2d37d1b203b2e82204e084b93f16819"} Jul 28 00:30:02.002872 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:02.002848 2555 generic.go:356] "Generic (PLEG): container finished" podID="983cabd4-fcab-4b6c-a2b0-d10b966c0b29" containerID="0ec1aca5d159ff2b36840be32c963eee912d577f011460d6f0c8930f4cd7df65" exitCode=0 Jul 28 00:30:02.002977 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:02.002924 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" event={"ID":"983cabd4-fcab-4b6c-a2b0-d10b966c0b29","Type":"ContainerDied","Data":"0ec1aca5d159ff2b36840be32c963eee912d577f011460d6f0c8930f4cd7df65"} Jul 28 00:30:02.004213 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:02.004193 2555 generic.go:356] "Generic (PLEG): container finished" podID="6142d31d-0d77-4b36-be26-8aa93752abc9" containerID="712122ab3b7496af541edf078d9541717013f896da9cbfbe69f7c7eebe77f9a6" exitCode=0 Jul 28 00:30:02.004310 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:02.004249 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" event={"ID":"6142d31d-0d77-4b36-be26-8aa93752abc9","Type":"ContainerDied","Data":"712122ab3b7496af541edf078d9541717013f896da9cbfbe69f7c7eebe77f9a6"} Jul 28 00:30:03.180867 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:03.180836 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" Jul 28 00:30:03.230330 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:03.230305 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" Jul 28 00:30:03.233700 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:03.233683 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" Jul 28 00:30:03.238085 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:03.238065 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" Jul 28 00:30:04.016459 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.016422 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" Jul 28 00:30:04.016685 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.016467 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2" event={"ID":"983cabd4-fcab-4b6c-a2b0-d10b966c0b29","Type":"ContainerDied","Data":"428bd1ed689b34ab91b2dc95af7d0711c90126c11e3ae219ebfc62f89231abd9"} Jul 28 00:30:04.016685 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.016530 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="428bd1ed689b34ab91b2dc95af7d0711c90126c11e3ae219ebfc62f89231abd9" Jul 28 00:30:04.017812 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.017793 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" Jul 28 00:30:04.017939 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.017835 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg" event={"ID":"6142d31d-0d77-4b36-be26-8aa93752abc9","Type":"ContainerDied","Data":"10e75c22e27cf3a6a7ce313d868d3c55330d10dbd37453667026e3da1d0564fc"} Jul 28 00:30:04.017939 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.017895 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e75c22e27cf3a6a7ce313d868d3c55330d10dbd37453667026e3da1d0564fc" Jul 28 00:30:04.019312 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.019295 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" Jul 28 00:30:04.019426 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.019325 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753310-nhjm5" event={"ID":"298f7354-fe28-4aeb-8a8d-21c389e24a78","Type":"ContainerDied","Data":"b936de0e394d4a358be7734a62f6773ac225a747655ab6f10471df9895e8960a"} Jul 28 00:30:04.019426 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.019371 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b936de0e394d4a358be7734a62f6773ac225a747655ab6f10471df9895e8960a" Jul 28 00:30:04.020841 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.020817 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" event={"ID":"66a1137c-1853-4d80-a4d8-e4b63361fbda","Type":"ContainerDied","Data":"5f281fe1c38f71aab8809bdaca5f6840ea5533ebab1c8a4b601b8faf72cfa289"} Jul 28 00:30:04.020951 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.020838 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r" Jul 28 00:30:04.020951 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:04.020856 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f281fe1c38f71aab8809bdaca5f6840ea5533ebab1c8a4b601b8faf72cfa289" Jul 28 00:30:09.846017 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:09.845980 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:09.947970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:09.947941 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:10.146470 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.146385 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:10.375379 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.375347 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:10.488168 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488083 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l"] Jul 28 00:30:10.488587 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488572 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6142d31d-0d77-4b36-be26-8aa93752abc9" containerName="cleanup" Jul 28 00:30:10.488650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488589 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6142d31d-0d77-4b36-be26-8aa93752abc9" containerName="cleanup" Jul 28 00:30:10.488650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488604 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298f7354-fe28-4aeb-8a8d-21c389e24a78" containerName="cleanup" Jul 28 00:30:10.488650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488609 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="298f7354-fe28-4aeb-8a8d-21c389e24a78" containerName="cleanup" Jul 28 00:30:10.488650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488621 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a1137c-1853-4d80-a4d8-e4b63361fbda" containerName="cleanup" Jul 28 00:30:10.488650 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488628 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a1137c-1853-4d80-a4d8-e4b63361fbda" containerName="cleanup" Jul 28 00:30:10.488874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488654 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983cabd4-fcab-4b6c-a2b0-d10b966c0b29" containerName="cleanup" Jul 28 00:30:10.488874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488663 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="983cabd4-fcab-4b6c-a2b0-d10b966c0b29" containerName="cleanup" Jul 28 00:30:10.488874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488759 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a1137c-1853-4d80-a4d8-e4b63361fbda" containerName="cleanup" Jul 28 00:30:10.488874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488772 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="983cabd4-fcab-4b6c-a2b0-d10b966c0b29" containerName="cleanup" Jul 28 00:30:10.488874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488782 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6142d31d-0d77-4b36-be26-8aa93752abc9" containerName="cleanup" Jul 28 00:30:10.488874 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.488795 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="298f7354-fe28-4aeb-8a8d-21c389e24a78" containerName="cleanup" Jul 28 00:30:10.492084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.492067 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:10.494090 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.494073 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:30:10.498166 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.498143 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l"] Jul 28 00:30:10.509322 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.509299 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e765bd3d-0c8c-4500-b794-d4d7dbff1850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l\" (UID: \"e765bd3d-0c8c-4500-b794-d4d7dbff1850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:10.610566 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.610528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e765bd3d-0c8c-4500-b794-d4d7dbff1850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l\" (UID: \"e765bd3d-0c8c-4500-b794-d4d7dbff1850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:10.611162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.611142 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e765bd3d-0c8c-4500-b794-d4d7dbff1850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l\" (UID: \"e765bd3d-0c8c-4500-b794-d4d7dbff1850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:10.805201 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.805166 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:10.939094 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:10.939065 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l"] Jul 28 00:30:10.941385 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:10.941353 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode765bd3d_0c8c_4500_b794_d4d7dbff1850.slice/crio-c1b1dd52dbe377342ebba16a480a7d6b42469cd1e8eea957e4839a04244c9cef WatchSource:0}: Error finding container c1b1dd52dbe377342ebba16a480a7d6b42469cd1e8eea957e4839a04244c9cef: Status 404 returned error can't find the container with id c1b1dd52dbe377342ebba16a480a7d6b42469cd1e8eea957e4839a04244c9cef Jul 28 00:30:11.059844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:11.059769 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" event={"ID":"e765bd3d-0c8c-4500-b794-d4d7dbff1850","Type":"ContainerStarted","Data":"c1b1dd52dbe377342ebba16a480a7d6b42469cd1e8eea957e4839a04244c9cef"} Jul 28 00:30:12.066589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:12.066558 2555 generic.go:356] "Generic (PLEG): container finished" podID="e765bd3d-0c8c-4500-b794-d4d7dbff1850" containerID="725bc512d802ce3ca901fa91890ffb4fbf03a2e992e693413822d2568d64a9c8" exitCode=0 Jul 28 00:30:12.067020 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:12.066650 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" event={"ID":"e765bd3d-0c8c-4500-b794-d4d7dbff1850","Type":"ContainerDied","Data":"725bc512d802ce3ca901fa91890ffb4fbf03a2e992e693413822d2568d64a9c8"} Jul 28 00:30:13.218590 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:13.218563 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:13.233276 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:13.233252 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e765bd3d-0c8c-4500-b794-d4d7dbff1850-maas-api-service-ca\") pod \"e765bd3d-0c8c-4500-b794-d4d7dbff1850\" (UID: \"e765bd3d-0c8c-4500-b794-d4d7dbff1850\") " Jul 28 00:30:13.233677 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:13.233650 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e765bd3d-0c8c-4500-b794-d4d7dbff1850-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e765bd3d-0c8c-4500-b794-d4d7dbff1850" (UID: "e765bd3d-0c8c-4500-b794-d4d7dbff1850"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:13.335191 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:13.335111 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e765bd3d-0c8c-4500-b794-d4d7dbff1850-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:14.078162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:14.078119 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" event={"ID":"e765bd3d-0c8c-4500-b794-d4d7dbff1850","Type":"ContainerDied","Data":"c1b1dd52dbe377342ebba16a480a7d6b42469cd1e8eea957e4839a04244c9cef"} Jul 28 00:30:14.078162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:14.078154 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l" Jul 28 00:30:14.078383 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:14.078180 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1b1dd52dbe377342ebba16a480a7d6b42469cd1e8eea957e4839a04244c9cef" Jul 28 00:30:22.276774 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.276738 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m"] Jul 28 00:30:22.277210 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.276921 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" podUID="5fd971cc-1ba2-4b09-8583-74e6d60bcc11" containerName="maas-api" containerID="cri-o://aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d" gracePeriod=30 Jul 28 00:30:22.300187 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.300161 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2"] Jul 28 00:30:22.302775 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.302753 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-056d1a-29753310-z7cn2"] Jul 28 00:30:22.303245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.303227 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:22.327977 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.327951 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l"] Jul 28 00:30:22.328173 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.328149 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" podUID="5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" containerName="payload-processing" containerID="cri-o://a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179" gracePeriod=30 Jul 28 00:30:22.335687 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.335654 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg"] Jul 28 00:30:22.335903 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.335876 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" podUID="df5ba36e-100d-4eca-8417-5000754a55ca" containerName="payload-pre-processing" containerID="cri-o://b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61" gracePeriod=30 Jul 28 00:30:22.368195 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.368157 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:22.469095 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.469004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:22.568334 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.568302 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983cabd4-fcab-4b6c-a2b0-d10b966c0b29" path="/var/lib/kubelet/pods/983cabd4-fcab-4b6c-a2b0-d10b966c0b29/volumes" Jul 28 00:30:22.619312 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.619290 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:30:22.627979 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.627948 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:30:22.637840 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.637814 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:30:22.719439 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719409 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-openshift-service-ca\") pod \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " Jul 28 00:30:22.719439 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719445 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ba36e-100d-4eca-8417-5000754a55ca-plugins-config-volume\") pod \"df5ba36e-100d-4eca-8417-5000754a55ca\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " Jul 28 00:30:22.719672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719468 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbk5c\" (UniqueName: \"kubernetes.io/projected/df5ba36e-100d-4eca-8417-5000754a55ca-kube-api-access-jbk5c\") pod \"df5ba36e-100d-4eca-8417-5000754a55ca\" (UID: \"df5ba36e-100d-4eca-8417-5000754a55ca\") " Jul 28 00:30:22.719672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719522 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-plugins-config-volume\") pod \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " Jul 28 00:30:22.719672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719553 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f764s\" (UniqueName: \"kubernetes.io/projected/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-kube-api-access-f764s\") pod \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " Jul 28 00:30:22.719672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719588 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqk8m\" (UniqueName: \"kubernetes.io/projected/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-kube-api-access-pqk8m\") pod \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\" (UID: \"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f\") " Jul 28 00:30:22.719672 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719626 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-maas-api-tls\") pod \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\" (UID: \"5fd971cc-1ba2-4b09-8583-74e6d60bcc11\") " Jul 28 00:30:22.719943 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719839 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fd971cc-1ba2-4b09-8583-74e6d60bcc11" (UID: "5fd971cc-1ba2-4b09-8583-74e6d60bcc11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:22.719943 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719871 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5ba36e-100d-4eca-8417-5000754a55ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df5ba36e-100d-4eca-8417-5000754a55ca" (UID: "df5ba36e-100d-4eca-8417-5000754a55ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:22.719943 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.719913 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" (UID: "5e06d7b1-abb2-4cbb-be02-4f09a33aa29f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:22.720066 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.720036 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:22.720066 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.720061 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5ba36e-100d-4eca-8417-5000754a55ca-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:22.720146 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.720075 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:22.721770 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.721739 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-kube-api-access-f764s" (OuterVolumeSpecName: "kube-api-access-f764s") pod "5fd971cc-1ba2-4b09-8583-74e6d60bcc11" (UID: "5fd971cc-1ba2-4b09-8583-74e6d60bcc11"). InnerVolumeSpecName "kube-api-access-f764s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:30:22.721890 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.721869 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-kube-api-access-pqk8m" (OuterVolumeSpecName: "kube-api-access-pqk8m") pod "5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" (UID: "5e06d7b1-abb2-4cbb-be02-4f09a33aa29f"). InnerVolumeSpecName "kube-api-access-pqk8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:30:22.722149 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.722124 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fd971cc-1ba2-4b09-8583-74e6d60bcc11" (UID: "5fd971cc-1ba2-4b09-8583-74e6d60bcc11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:30:22.722149 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.722137 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5ba36e-100d-4eca-8417-5000754a55ca-kube-api-access-jbk5c" (OuterVolumeSpecName: "kube-api-access-jbk5c") pod "df5ba36e-100d-4eca-8417-5000754a55ca" (UID: "df5ba36e-100d-4eca-8417-5000754a55ca"). InnerVolumeSpecName "kube-api-access-jbk5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:30:22.821531 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.821453 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbk5c\" (UniqueName: \"kubernetes.io/projected/df5ba36e-100d-4eca-8417-5000754a55ca-kube-api-access-jbk5c\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:22.821531 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.821476 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f764s\" (UniqueName: \"kubernetes.io/projected/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-kube-api-access-f764s\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:22.821531 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.821486 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqk8m\" (UniqueName: \"kubernetes.io/projected/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f-kube-api-access-pqk8m\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:22.821531 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:22.821521 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd971cc-1ba2-4b09-8583-74e6d60bcc11-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:23.124581 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.124477 2555 generic.go:356] "Generic (PLEG): container finished" podID="5fd971cc-1ba2-4b09-8583-74e6d60bcc11" containerID="aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d" exitCode=0 Jul 28 00:30:23.124581 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.124539 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" Jul 28 00:30:23.124581 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.124571 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" event={"ID":"5fd971cc-1ba2-4b09-8583-74e6d60bcc11","Type":"ContainerDied","Data":"aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d"} Jul 28 00:30:23.124819 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.124628 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m" event={"ID":"5fd971cc-1ba2-4b09-8583-74e6d60bcc11","Type":"ContainerDied","Data":"89a90100b57a09acfcaa60df8bff63367fdc518882bfa1da707dfb012c2ecea0"} Jul 28 00:30:23.124819 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.124643 2555 scope.go:117] "RemoveContainer" containerID="aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d" Jul 28 00:30:23.126029 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.126008 2555 generic.go:356] "Generic (PLEG): container finished" podID="df5ba36e-100d-4eca-8417-5000754a55ca" containerID="b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61" exitCode=0 Jul 28 00:30:23.126132 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.126032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" event={"ID":"df5ba36e-100d-4eca-8417-5000754a55ca","Type":"ContainerDied","Data":"b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61"} Jul 28 00:30:23.126132 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.126070 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" Jul 28 00:30:23.126132 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.126083 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg" event={"ID":"df5ba36e-100d-4eca-8417-5000754a55ca","Type":"ContainerDied","Data":"a1be33616c3fad148225bf4ac76385a07f066452ae7761923cc55832e50e928c"} Jul 28 00:30:23.129416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.129332 2555 generic.go:356] "Generic (PLEG): container finished" podID="5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" containerID="a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179" exitCode=0 Jul 28 00:30:23.129642 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.129621 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" event={"ID":"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f","Type":"ContainerDied","Data":"a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179"} Jul 28 00:30:23.129729 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.129714 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" event={"ID":"5e06d7b1-abb2-4cbb-be02-4f09a33aa29f","Type":"ContainerDied","Data":"3cc9429b51134fe93dc56e0a8daa03f734a1cdb8d53af8e14f41590b5c9b374d"} Jul 28 00:30:23.129821 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.129799 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l" Jul 28 00:30:23.137323 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.137300 2555 scope.go:117] "RemoveContainer" containerID="aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d" Jul 28 00:30:23.137895 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:30:23.137865 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d\": container with ID starting with aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d not found: ID does not exist" containerID="aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d" Jul 28 00:30:23.137980 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.137908 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d"} err="failed to get container status \"aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d\": rpc error: code = NotFound desc = could not find container \"aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d\": container with ID starting with aed6e52293371f4b5472445ed9c9bed64a6babdf50097b0c9910db3019d9871d not found: ID does not exist" Jul 28 00:30:23.137980 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.137939 2555 scope.go:117] "RemoveContainer" containerID="b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61" Jul 28 00:30:23.148213 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.148195 2555 scope.go:117] "RemoveContainer" containerID="b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61" Jul 28 00:30:23.148465 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:30:23.148440 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61\": container with ID starting with b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61 not found: ID does not exist" containerID="b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61" Jul 28 00:30:23.148603 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.148476 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61"} err="failed to get container status \"b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61\": rpc error: code = NotFound desc = could not find container \"b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61\": container with ID starting with b5926a46bc50cdc8511e98c5731c8a1dcb17b688171533f1f075dd5e8d170d61 not found: ID does not exist" Jul 28 00:30:23.148603 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.148522 2555 scope.go:117] "RemoveContainer" containerID="a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179" Jul 28 00:30:23.153150 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.153131 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m"] Jul 28 00:30:23.157203 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.157187 2555 scope.go:117] "RemoveContainer" containerID="a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179" Jul 28 00:30:23.157268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.157215 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-056d1a-dcb8495f9-4vm6m"] Jul 28 00:30:23.157435 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:30:23.157417 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179\": container with ID starting with a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179 not found: ID does not exist" containerID="a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179" Jul 28 00:30:23.157482 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.157441 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179"} err="failed to get container status \"a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179\": rpc error: code = NotFound desc = could not find container \"a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179\": container with ID starting with a926d78f08f20efe70a7f2b973a14f14d59faa63255577cc8da2516a173cd179 not found: ID does not exist" Jul 28 00:30:23.168292 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.168274 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg"] Jul 28 00:30:23.173752 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.173730 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056d1a-7f8b679896-48hrg"] Jul 28 00:30:23.182795 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.182773 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l"] Jul 28 00:30:23.186667 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:23.186649 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056d1a-d4bf5d966-5z69l"] Jul 28 00:30:24.566008 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:24.565976 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" path="/var/lib/kubelet/pods/5e06d7b1-abb2-4cbb-be02-4f09a33aa29f/volumes" Jul 28 00:30:24.566365 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:24.566341 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd971cc-1ba2-4b09-8583-74e6d60bcc11" path="/var/lib/kubelet/pods/5fd971cc-1ba2-4b09-8583-74e6d60bcc11/volumes" Jul 28 00:30:24.566682 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:24.566669 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5ba36e-100d-4eca-8417-5000754a55ca" path="/var/lib/kubelet/pods/df5ba36e-100d-4eca-8417-5000754a55ca/volumes" Jul 28 00:30:25.567406 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:25.567374 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:25.573591 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:25.573562 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l"] Jul 28 00:30:25.577904 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:25.577880 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-056d1a-h5c7l"] Jul 28 00:30:26.039510 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:26.039473 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:26.565575 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:26.565538 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e765bd3d-0c8c-4500-b794-d4d7dbff1850" path="/var/lib/kubelet/pods/e765bd3d-0c8c-4500-b794-d4d7dbff1850/volumes" Jul 28 00:30:32.162459 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.162430 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:32.262646 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.262612 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:32.437693 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.437610 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:32.688386 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.688299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:32.826362 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826276 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj"] Jul 28 00:30:32.826805 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826785 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e765bd3d-0c8c-4500-b794-d4d7dbff1850" containerName="revoke-keys" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826806 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e765bd3d-0c8c-4500-b794-d4d7dbff1850" containerName="revoke-keys" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826820 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" containerName="payload-processing" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826828 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" containerName="payload-processing" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826846 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5ba36e-100d-4eca-8417-5000754a55ca" containerName="payload-pre-processing" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826856 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5ba36e-100d-4eca-8417-5000754a55ca" containerName="payload-pre-processing" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826867 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd971cc-1ba2-4b09-8583-74e6d60bcc11" containerName="maas-api" Jul 28 00:30:32.826908 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826876 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd971cc-1ba2-4b09-8583-74e6d60bcc11" containerName="maas-api" Jul 28 00:30:32.827244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.826998 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e765bd3d-0c8c-4500-b794-d4d7dbff1850" containerName="revoke-keys" Jul 28 00:30:32.827244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.827012 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e06d7b1-abb2-4cbb-be02-4f09a33aa29f" containerName="payload-processing" Jul 28 00:30:32.827244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.827023 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5ba36e-100d-4eca-8417-5000754a55ca" containerName="payload-pre-processing" Jul 28 00:30:32.827244 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.827035 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd971cc-1ba2-4b09-8583-74e6d60bcc11" containerName="maas-api" Jul 28 00:30:32.831397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.831375 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:32.833409 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.833388 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:30:32.837660 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.837635 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj"] Jul 28 00:30:32.912411 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:32.912378 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e50ded1-7fad-484c-9bf5-78bbf550e749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj\" (UID: \"5e50ded1-7fad-484c-9bf5-78bbf550e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:33.013624 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:33.013588 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e50ded1-7fad-484c-9bf5-78bbf550e749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj\" (UID: \"5e50ded1-7fad-484c-9bf5-78bbf550e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:33.014256 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:33.014231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e50ded1-7fad-484c-9bf5-78bbf550e749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj\" (UID: \"5e50ded1-7fad-484c-9bf5-78bbf550e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:33.145729 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:33.145690 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:33.305088 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:33.305061 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj"] Jul 28 00:30:33.308347 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:33.308316 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e50ded1_7fad_484c_9bf5_78bbf550e749.slice/crio-ad3fd53f4ecd2305ce7135cdd486230a37f2e2ac2762413c2727e9d2b35af7bc WatchSource:0}: Error finding container ad3fd53f4ecd2305ce7135cdd486230a37f2e2ac2762413c2727e9d2b35af7bc: Status 404 returned error can't find the container with id ad3fd53f4ecd2305ce7135cdd486230a37f2e2ac2762413c2727e9d2b35af7bc Jul 28 00:30:34.195460 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:34.195424 2555 generic.go:356] "Generic (PLEG): container finished" podID="5e50ded1-7fad-484c-9bf5-78bbf550e749" containerID="60d117ced64d8514fd749490e439354a62fe50d7c9cbcf4475312e3647e6a555" exitCode=0 Jul 28 00:30:34.195636 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:34.195487 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" event={"ID":"5e50ded1-7fad-484c-9bf5-78bbf550e749","Type":"ContainerDied","Data":"60d117ced64d8514fd749490e439354a62fe50d7c9cbcf4475312e3647e6a555"} Jul 28 00:30:34.195636 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:34.195570 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" event={"ID":"5e50ded1-7fad-484c-9bf5-78bbf550e749","Type":"ContainerStarted","Data":"ad3fd53f4ecd2305ce7135cdd486230a37f2e2ac2762413c2727e9d2b35af7bc"} Jul 28 00:30:35.337182 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:35.337155 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:35.437171 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:35.437140 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e50ded1-7fad-484c-9bf5-78bbf550e749-maas-api-service-ca\") pod \"5e50ded1-7fad-484c-9bf5-78bbf550e749\" (UID: \"5e50ded1-7fad-484c-9bf5-78bbf550e749\") " Jul 28 00:30:35.437491 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:35.437465 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e50ded1-7fad-484c-9bf5-78bbf550e749-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e50ded1-7fad-484c-9bf5-78bbf550e749" (UID: "5e50ded1-7fad-484c-9bf5-78bbf550e749"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:35.537797 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:35.537767 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e50ded1-7fad-484c-9bf5-78bbf550e749-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:36.207811 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:36.207776 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" Jul 28 00:30:36.207990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:36.207813 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj" event={"ID":"5e50ded1-7fad-484c-9bf5-78bbf550e749","Type":"ContainerDied","Data":"ad3fd53f4ecd2305ce7135cdd486230a37f2e2ac2762413c2727e9d2b35af7bc"} Jul 28 00:30:36.207990 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:36.207862 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3fd53f4ecd2305ce7135cdd486230a37f2e2ac2762413c2727e9d2b35af7bc" Jul 28 00:30:42.975814 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:42.975760 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7"] Jul 28 00:30:42.976434 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:42.976004 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" podUID="e9f224be-5549-477b-ac38-522232c4c761" containerName="maas-api" containerID="cri-o://f3be4bcc0fdc61d34240548b94e0c3c65a1f0bce5fba44f21dea81f19b7a1827" gracePeriod=30 Jul 28 00:30:43.005669 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.005643 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg"] Jul 28 00:30:43.008682 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.008657 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-328ef1-29753310-7q2vg"] Jul 28 00:30:43.014342 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.014312 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:43.033413 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.033385 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg"] Jul 28 00:30:43.033647 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.033616 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" podUID="335792df-10dd-4882-81f1-6112414e6631" containerName="payload-processing" containerID="cri-o://cff24721d92707a7ac8438621a2cdf83b462077ff4c9afdb1e6848e118bd4c16" gracePeriod=30 Jul 28 00:30:43.039753 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.039721 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg"] Jul 28 00:30:43.039959 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.039930 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" podUID="858d61d2-540f-49d6-864a-664fc1a72708" containerName="payload-pre-processing" containerID="cri-o://effefece0f038c77ed2d713fe366dbfe87a03f89972947cbe0e9517434735638" gracePeriod=30 Jul 28 00:30:43.077106 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.077077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:43.178757 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.178714 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:43.253199 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.253173 2555 generic.go:356] "Generic (PLEG): container finished" podID="e9f224be-5549-477b-ac38-522232c4c761" containerID="f3be4bcc0fdc61d34240548b94e0c3c65a1f0bce5fba44f21dea81f19b7a1827" exitCode=0 Jul 28 00:30:43.253336 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.253256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" event={"ID":"e9f224be-5549-477b-ac38-522232c4c761","Type":"ContainerDied","Data":"f3be4bcc0fdc61d34240548b94e0c3c65a1f0bce5fba44f21dea81f19b7a1827"} Jul 28 00:30:43.255023 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.254998 2555 generic.go:356] "Generic (PLEG): container finished" podID="858d61d2-540f-49d6-864a-664fc1a72708" containerID="effefece0f038c77ed2d713fe366dbfe87a03f89972947cbe0e9517434735638" exitCode=0 Jul 28 00:30:43.255161 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.255065 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" event={"ID":"858d61d2-540f-49d6-864a-664fc1a72708","Type":"ContainerDied","Data":"effefece0f038c77ed2d713fe366dbfe87a03f89972947cbe0e9517434735638"} Jul 28 00:30:43.271194 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.270991 2555 generic.go:356] "Generic (PLEG): container finished" podID="335792df-10dd-4882-81f1-6112414e6631" containerID="cff24721d92707a7ac8438621a2cdf83b462077ff4c9afdb1e6848e118bd4c16" exitCode=0 Jul 28 00:30:43.271194 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.271027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" event={"ID":"335792df-10dd-4882-81f1-6112414e6631","Type":"ContainerDied","Data":"cff24721d92707a7ac8438621a2cdf83b462077ff4c9afdb1e6848e118bd4c16"} Jul 28 00:30:43.309254 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.309221 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:30:43.339227 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.339200 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:30:43.351613 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.351591 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:30:43.405256 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.405227 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f224be-5549-477b-ac38-522232c4c761-openshift-service-ca\") pod \"e9f224be-5549-477b-ac38-522232c4c761\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " Jul 28 00:30:43.405416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.405312 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klj7z\" (UniqueName: \"kubernetes.io/projected/e9f224be-5549-477b-ac38-522232c4c761-kube-api-access-klj7z\") pod \"e9f224be-5549-477b-ac38-522232c4c761\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " Jul 28 00:30:43.405416 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.405378 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f224be-5549-477b-ac38-522232c4c761-maas-api-tls\") pod \"e9f224be-5549-477b-ac38-522232c4c761\" (UID: \"e9f224be-5549-477b-ac38-522232c4c761\") " Jul 28 00:30:43.405603 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.405582 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f224be-5549-477b-ac38-522232c4c761-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9f224be-5549-477b-ac38-522232c4c761" (UID: "e9f224be-5549-477b-ac38-522232c4c761"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:43.407368 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.407346 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f224be-5549-477b-ac38-522232c4c761-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9f224be-5549-477b-ac38-522232c4c761" (UID: "e9f224be-5549-477b-ac38-522232c4c761"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:30:43.407452 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.407387 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f224be-5549-477b-ac38-522232c4c761-kube-api-access-klj7z" (OuterVolumeSpecName: "kube-api-access-klj7z") pod "e9f224be-5549-477b-ac38-522232c4c761" (UID: "e9f224be-5549-477b-ac38-522232c4c761"). InnerVolumeSpecName "kube-api-access-klj7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:30:43.506649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.506572 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqjsl\" (UniqueName: \"kubernetes.io/projected/858d61d2-540f-49d6-864a-664fc1a72708-kube-api-access-lqjsl\") pod \"858d61d2-540f-49d6-864a-664fc1a72708\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " Jul 28 00:30:43.506649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.506631 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdb7x\" (UniqueName: \"kubernetes.io/projected/335792df-10dd-4882-81f1-6112414e6631-kube-api-access-bdb7x\") pod \"335792df-10dd-4882-81f1-6112414e6631\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " Jul 28 00:30:43.506649 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.506650 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858d61d2-540f-49d6-864a-664fc1a72708-plugins-config-volume\") pod \"858d61d2-540f-49d6-864a-664fc1a72708\" (UID: \"858d61d2-540f-49d6-864a-664fc1a72708\") " Jul 28 00:30:43.506916 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.506782 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335792df-10dd-4882-81f1-6112414e6631-plugins-config-volume\") pod \"335792df-10dd-4882-81f1-6112414e6631\" (UID: \"335792df-10dd-4882-81f1-6112414e6631\") " Jul 28 00:30:43.507110 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.507089 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj7z\" (UniqueName: \"kubernetes.io/projected/e9f224be-5549-477b-ac38-522232c4c761-kube-api-access-klj7z\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:43.507210 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.507115 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f224be-5549-477b-ac38-522232c4c761-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:43.507210 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.507125 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f224be-5549-477b-ac38-522232c4c761-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:43.507210 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.507089 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858d61d2-540f-49d6-864a-664fc1a72708-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "858d61d2-540f-49d6-864a-664fc1a72708" (UID: "858d61d2-540f-49d6-864a-664fc1a72708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:43.507210 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.507125 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335792df-10dd-4882-81f1-6112414e6631-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "335792df-10dd-4882-81f1-6112414e6631" (UID: "335792df-10dd-4882-81f1-6112414e6631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:43.508736 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.508715 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858d61d2-540f-49d6-864a-664fc1a72708-kube-api-access-lqjsl" (OuterVolumeSpecName: "kube-api-access-lqjsl") pod "858d61d2-540f-49d6-864a-664fc1a72708" (UID: "858d61d2-540f-49d6-864a-664fc1a72708"). InnerVolumeSpecName "kube-api-access-lqjsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:30:43.509050 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.509030 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335792df-10dd-4882-81f1-6112414e6631-kube-api-access-bdb7x" (OuterVolumeSpecName: "kube-api-access-bdb7x") pod "335792df-10dd-4882-81f1-6112414e6631" (UID: "335792df-10dd-4882-81f1-6112414e6631"). InnerVolumeSpecName "kube-api-access-bdb7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:30:43.607878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.607838 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjsl\" (UniqueName: \"kubernetes.io/projected/858d61d2-540f-49d6-864a-664fc1a72708-kube-api-access-lqjsl\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:43.607878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.607870 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdb7x\" (UniqueName: \"kubernetes.io/projected/335792df-10dd-4882-81f1-6112414e6631-kube-api-access-bdb7x\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:43.607878 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.607880 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858d61d2-540f-49d6-864a-664fc1a72708-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:43.608107 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:43.607889 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335792df-10dd-4882-81f1-6112414e6631-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:44.277089 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.277008 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" Jul 28 00:30:44.277089 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.277055 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg" event={"ID":"335792df-10dd-4882-81f1-6112414e6631","Type":"ContainerDied","Data":"13e05cb8aa2f2c2c0b2f753e03e998795328701ef7defcb2b6399182ae1a48bc"} Jul 28 00:30:44.277622 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.277113 2555 scope.go:117] "RemoveContainer" containerID="cff24721d92707a7ac8438621a2cdf83b462077ff4c9afdb1e6848e118bd4c16" Jul 28 00:30:44.278809 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.278792 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" Jul 28 00:30:44.278931 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.278822 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7" event={"ID":"e9f224be-5549-477b-ac38-522232c4c761","Type":"ContainerDied","Data":"73bf5bdb4bb3d500bfedb7a4ac2d84367c8354501c0295b7dc624ddd461cebea"} Jul 28 00:30:44.280780 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.280758 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" event={"ID":"858d61d2-540f-49d6-864a-664fc1a72708","Type":"ContainerDied","Data":"9d875a03700d651b4aa2254c2ef069dc31359a83893a92cfb5effd24326487f5"} Jul 28 00:30:44.280780 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.280776 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg" Jul 28 00:30:44.286640 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.286574 2555 scope.go:117] "RemoveContainer" containerID="f3be4bcc0fdc61d34240548b94e0c3c65a1f0bce5fba44f21dea81f19b7a1827" Jul 28 00:30:44.295559 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.295543 2555 scope.go:117] "RemoveContainer" containerID="effefece0f038c77ed2d713fe366dbfe87a03f89972947cbe0e9517434735638" Jul 28 00:30:44.303172 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.303148 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg"] Jul 28 00:30:44.306332 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.306314 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-328ef1-b9d99c778-f4sdg"] Jul 28 00:30:44.314589 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.314570 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7"] Jul 28 00:30:44.317931 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.317912 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-328ef1-5657cf8cdd-l9dd7"] Jul 28 00:30:44.327246 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.327225 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg"] Jul 28 00:30:44.330370 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.330352 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-328ef1-7454b4f444-pstkg"] Jul 28 00:30:44.565384 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.565292 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335792df-10dd-4882-81f1-6112414e6631" path="/var/lib/kubelet/pods/335792df-10dd-4882-81f1-6112414e6631/volumes" Jul 28 00:30:44.565627 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.565609 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6142d31d-0d77-4b36-be26-8aa93752abc9" path="/var/lib/kubelet/pods/6142d31d-0d77-4b36-be26-8aa93752abc9/volumes" Jul 28 00:30:44.565909 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.565896 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858d61d2-540f-49d6-864a-664fc1a72708" path="/var/lib/kubelet/pods/858d61d2-540f-49d6-864a-664fc1a72708/volumes" Jul 28 00:30:44.566232 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:44.566221 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f224be-5549-477b-ac38-522232c4c761" path="/var/lib/kubelet/pods/e9f224be-5549-477b-ac38-522232c4c761/volumes" Jul 28 00:30:47.896441 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:47.896405 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:47.906651 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:47.906626 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj"] Jul 28 00:30:47.916769 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:47.911004 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-328ef1-vffgj"] Jul 28 00:30:48.302583 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:48.302550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:48.565655 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:48.565570 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e50ded1-7fad-484c-9bf5-78bbf550e749" path="/var/lib/kubelet/pods/5e50ded1-7fad-484c-9bf5-78bbf550e749/volumes" Jul 28 00:30:53.987733 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:53.987695 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:54.087802 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.087771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:54.255641 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.255616 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:54.489687 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.489655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:30:54.616800 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.616709 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt"] Jul 28 00:30:54.617177 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617158 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335792df-10dd-4882-81f1-6112414e6631" containerName="payload-processing" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617181 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="335792df-10dd-4882-81f1-6112414e6631" containerName="payload-processing" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617208 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858d61d2-540f-49d6-864a-664fc1a72708" containerName="payload-pre-processing" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617217 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="858d61d2-540f-49d6-864a-664fc1a72708" containerName="payload-pre-processing" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617238 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f224be-5549-477b-ac38-522232c4c761" containerName="maas-api" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617247 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f224be-5549-477b-ac38-522232c4c761" containerName="maas-api" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617258 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e50ded1-7fad-484c-9bf5-78bbf550e749" containerName="revoke-keys" Jul 28 00:30:54.617268 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617265 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e50ded1-7fad-484c-9bf5-78bbf550e749" containerName="revoke-keys" Jul 28 00:30:54.617637 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617354 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f224be-5549-477b-ac38-522232c4c761" containerName="maas-api" Jul 28 00:30:54.617637 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617366 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="335792df-10dd-4882-81f1-6112414e6631" containerName="payload-processing" Jul 28 00:30:54.617637 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617377 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e50ded1-7fad-484c-9bf5-78bbf550e749" containerName="revoke-keys" Jul 28 00:30:54.617637 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.617389 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="858d61d2-540f-49d6-864a-664fc1a72708" containerName="payload-pre-processing" Jul 28 00:30:54.620537 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.620518 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:54.622469 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.622451 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9jgn\"" Jul 28 00:30:54.626510 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.626469 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt"] Jul 28 00:30:54.804021 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.803974 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52765958-1ba9-4fd7-9206-b474dbe5e266-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt\" (UID: \"52765958-1ba9-4fd7-9206-b474dbe5e266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:54.904978 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.904891 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52765958-1ba9-4fd7-9206-b474dbe5e266-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt\" (UID: \"52765958-1ba9-4fd7-9206-b474dbe5e266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:54.905489 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.905469 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52765958-1ba9-4fd7-9206-b474dbe5e266-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt\" (UID: \"52765958-1ba9-4fd7-9206-b474dbe5e266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:54.932883 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:54.932858 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:55.285459 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:55.285434 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt"] Jul 28 00:30:55.287611 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:30:55.287575 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52765958_1ba9_4fd7_9206_b474dbe5e266.slice/crio-f5e5e9385db8d8c4a61dfde5a192db1efabb24e3c86363a456583e254034d4ac WatchSource:0}: Error finding container f5e5e9385db8d8c4a61dfde5a192db1efabb24e3c86363a456583e254034d4ac: Status 404 returned error can't find the container with id f5e5e9385db8d8c4a61dfde5a192db1efabb24e3c86363a456583e254034d4ac Jul 28 00:30:55.340926 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:55.340878 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" event={"ID":"52765958-1ba9-4fd7-9206-b474dbe5e266","Type":"ContainerStarted","Data":"f5e5e9385db8d8c4a61dfde5a192db1efabb24e3c86363a456583e254034d4ac"} Jul 28 00:30:56.347848 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:56.347816 2555 generic.go:356] "Generic (PLEG): container finished" podID="52765958-1ba9-4fd7-9206-b474dbe5e266" containerID="66f5befb9fe9b9248cbe51ce9d267890d2c811d0d1170406af4279773ea80201" exitCode=0 Jul 28 00:30:56.348223 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:56.347882 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" event={"ID":"52765958-1ba9-4fd7-9206-b474dbe5e266","Type":"ContainerDied","Data":"66f5befb9fe9b9248cbe51ce9d267890d2c811d0d1170406af4279773ea80201"} Jul 28 00:30:57.485217 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:57.485195 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:57.525747 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:57.525718 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52765958-1ba9-4fd7-9206-b474dbe5e266-maas-api-service-ca\") pod \"52765958-1ba9-4fd7-9206-b474dbe5e266\" (UID: \"52765958-1ba9-4fd7-9206-b474dbe5e266\") " Jul 28 00:30:57.526086 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:57.526067 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52765958-1ba9-4fd7-9206-b474dbe5e266-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52765958-1ba9-4fd7-9206-b474dbe5e266" (UID: "52765958-1ba9-4fd7-9206-b474dbe5e266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:30:57.626664 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:57.626560 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52765958-1ba9-4fd7-9206-b474dbe5e266-maas-api-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:30:58.358979 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:58.358946 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" Jul 28 00:30:58.358979 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:58.358969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt" event={"ID":"52765958-1ba9-4fd7-9206-b474dbe5e266","Type":"ContainerDied","Data":"f5e5e9385db8d8c4a61dfde5a192db1efabb24e3c86363a456583e254034d4ac"} Jul 28 00:30:58.359198 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:30:58.359022 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e5e9385db8d8c4a61dfde5a192db1efabb24e3c86363a456583e254034d4ac" Jul 28 00:31:01.104777 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:01.104741 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:04.998435 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:04.998393 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx"] Jul 28 00:31:04.998945 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:04.998703 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" podUID="842e7ea4-32f1-4418-adf3-498ac4383318" containerName="maas-api" containerID="cri-o://5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5" gracePeriod=30 Jul 28 00:31:05.027569 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.027539 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r"] Jul 28 00:31:05.030391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.030364 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-6e175a-29753310-jxd6r"] Jul 28 00:31:05.031235 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.031211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:05.113941 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.113907 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k"] Jul 28 00:31:05.114197 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.114169 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" podUID="91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" containerName="payload-processing" containerID="cri-o://40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c" gracePeriod=30 Jul 28 00:31:05.119857 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.119825 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms"] Jul 28 00:31:05.120094 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.120064 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" podUID="0a18cc2b-4f56-48d2-8cc5-dace579cfdad" containerName="payload-pre-processing" containerID="cri-o://4b622634ebb8ef959b407ee21a805e2f6305f737bcf1594ea6970a079744b876" gracePeriod=30 Jul 28 00:31:05.148652 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.148618 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:05.281815 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.281784 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:31:05.388530 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.388511 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:31:05.394570 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.394547 2555 generic.go:356] "Generic (PLEG): container finished" podID="0a18cc2b-4f56-48d2-8cc5-dace579cfdad" containerID="4b622634ebb8ef959b407ee21a805e2f6305f737bcf1594ea6970a079744b876" exitCode=0 Jul 28 00:31:05.394655 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.394623 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" event={"ID":"0a18cc2b-4f56-48d2-8cc5-dace579cfdad","Type":"ContainerDied","Data":"4b622634ebb8ef959b407ee21a805e2f6305f737bcf1594ea6970a079744b876"} Jul 28 00:31:05.394704 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.394672 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" event={"ID":"0a18cc2b-4f56-48d2-8cc5-dace579cfdad","Type":"ContainerDied","Data":"dab9efb1e129667eb6887d8207fd6c80dbf11c58d2b6b975b407f4b278d199f2"} Jul 28 00:31:05.394753 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.394700 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dab9efb1e129667eb6887d8207fd6c80dbf11c58d2b6b975b407f4b278d199f2" Jul 28 00:31:05.395176 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.395160 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:31:05.395847 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.395830 2555 generic.go:356] "Generic (PLEG): container finished" podID="842e7ea4-32f1-4418-adf3-498ac4383318" containerID="5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5" exitCode=0 Jul 28 00:31:05.395915 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.395866 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" Jul 28 00:31:05.395957 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.395915 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" event={"ID":"842e7ea4-32f1-4418-adf3-498ac4383318","Type":"ContainerDied","Data":"5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5"} Jul 28 00:31:05.396006 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.395963 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx" event={"ID":"842e7ea4-32f1-4418-adf3-498ac4383318","Type":"ContainerDied","Data":"fba2e78249207b32ca25e84f5c1962e2c4a95df791013dfa385b1c6d107c97bb"} Jul 28 00:31:05.396058 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.395998 2555 scope.go:117] "RemoveContainer" containerID="5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5" Jul 28 00:31:05.397102 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.397086 2555 generic.go:356] "Generic (PLEG): container finished" podID="91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" containerID="40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c" exitCode=0 Jul 28 00:31:05.397154 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.397125 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" event={"ID":"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe","Type":"ContainerDied","Data":"40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c"} Jul 28 00:31:05.397154 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.397121 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" Jul 28 00:31:05.397154 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.397150 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k" event={"ID":"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe","Type":"ContainerDied","Data":"8cbc35199322410cb0217079bd70d0d2c524cbea42a8b330cc1c6f95b1f5323b"} Jul 28 00:31:05.401355 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.401336 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgb5m\" (UniqueName: \"kubernetes.io/projected/842e7ea4-32f1-4418-adf3-498ac4383318-kube-api-access-lgb5m\") pod \"842e7ea4-32f1-4418-adf3-498ac4383318\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " Jul 28 00:31:05.401436 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.401382 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842e7ea4-32f1-4418-adf3-498ac4383318-openshift-service-ca\") pod \"842e7ea4-32f1-4418-adf3-498ac4383318\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " Jul 28 00:31:05.401436 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.401429 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842e7ea4-32f1-4418-adf3-498ac4383318-maas-api-tls\") pod \"842e7ea4-32f1-4418-adf3-498ac4383318\" (UID: \"842e7ea4-32f1-4418-adf3-498ac4383318\") " Jul 28 00:31:05.401780 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.401760 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842e7ea4-32f1-4418-adf3-498ac4383318-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "842e7ea4-32f1-4418-adf3-498ac4383318" (UID: "842e7ea4-32f1-4418-adf3-498ac4383318"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:31:05.403742 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.403444 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842e7ea4-32f1-4418-adf3-498ac4383318-kube-api-access-lgb5m" (OuterVolumeSpecName: "kube-api-access-lgb5m") pod "842e7ea4-32f1-4418-adf3-498ac4383318" (UID: "842e7ea4-32f1-4418-adf3-498ac4383318"). InnerVolumeSpecName "kube-api-access-lgb5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:31:05.404215 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.404186 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842e7ea4-32f1-4418-adf3-498ac4383318-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "842e7ea4-32f1-4418-adf3-498ac4383318" (UID: "842e7ea4-32f1-4418-adf3-498ac4383318"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:31:05.405484 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.405387 2555 scope.go:117] "RemoveContainer" containerID="5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5" Jul 28 00:31:05.405716 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:31:05.405688 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5\": container with ID starting with 5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5 not found: ID does not exist" containerID="5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5" Jul 28 00:31:05.405929 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.405729 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5"} err="failed to get container status \"5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5\": rpc error: code = NotFound desc = could not find container \"5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5\": container with ID starting with 5cacc275f458c489a5581b390e45e7e2f8823edeef4b543094b5135768f3cdb5 not found: ID does not exist" Jul 28 00:31:05.405929 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.405768 2555 scope.go:117] "RemoveContainer" containerID="40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c" Jul 28 00:31:05.414666 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.414646 2555 scope.go:117] "RemoveContainer" containerID="40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c" Jul 28 00:31:05.414939 ip-10-0-130-94 kubenswrapper[2555]: E0728 00:31:05.414919 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c\": container with ID starting with 40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c not found: ID does not exist" containerID="40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c" Jul 28 00:31:05.415015 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.414948 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c"} err="failed to get container status \"40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c\": rpc error: code = NotFound desc = could not find container \"40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c\": container with ID starting with 40d5fda4c188618d0d7b5ad03b85a95c8b4ae3a2a891b54af5f00b8457f4572c not found: ID does not exist" Jul 28 00:31:05.502357 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502330 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-plugins-config-volume\") pod \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " Jul 28 00:31:05.502542 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502366 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-plugins-config-volume\") pod \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " Jul 28 00:31:05.502542 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502391 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2lsc\" (UniqueName: \"kubernetes.io/projected/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-kube-api-access-j2lsc\") pod \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\" (UID: \"0a18cc2b-4f56-48d2-8cc5-dace579cfdad\") " Jul 28 00:31:05.502542 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502410 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc28t\" (UniqueName: \"kubernetes.io/projected/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-kube-api-access-sc28t\") pod \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\" (UID: \"91bf45cf-c03b-4e10-8f45-c46b4d25cdbe\") " Jul 28 00:31:05.502661 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502648 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgb5m\" (UniqueName: \"kubernetes.io/projected/842e7ea4-32f1-4418-adf3-498ac4383318-kube-api-access-lgb5m\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.502698 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502670 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842e7ea4-32f1-4418-adf3-498ac4383318-openshift-service-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.502698 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502684 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842e7ea4-32f1-4418-adf3-498ac4383318-maas-api-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.502844 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502814 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" (UID: "91bf45cf-c03b-4e10-8f45-c46b4d25cdbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:31:05.502942 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.502812 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a18cc2b-4f56-48d2-8cc5-dace579cfdad" (UID: "0a18cc2b-4f56-48d2-8cc5-dace579cfdad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:31:05.504442 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.504422 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-kube-api-access-sc28t" (OuterVolumeSpecName: "kube-api-access-sc28t") pod "91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" (UID: "91bf45cf-c03b-4e10-8f45-c46b4d25cdbe"). InnerVolumeSpecName "kube-api-access-sc28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:31:05.504539 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.504451 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-kube-api-access-j2lsc" (OuterVolumeSpecName: "kube-api-access-j2lsc") pod "0a18cc2b-4f56-48d2-8cc5-dace579cfdad" (UID: "0a18cc2b-4f56-48d2-8cc5-dace579cfdad"). InnerVolumeSpecName "kube-api-access-j2lsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:31:05.603684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.603606 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2lsc\" (UniqueName: \"kubernetes.io/projected/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-kube-api-access-j2lsc\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.603684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.603634 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc28t\" (UniqueName: \"kubernetes.io/projected/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-kube-api-access-sc28t\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.603684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.603645 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.603684 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.603654 2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a18cc2b-4f56-48d2-8cc5-dace579cfdad-plugins-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jul 28 00:31:05.719390 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.719358 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k"] Jul 28 00:31:05.722364 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.722338 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-6e175a-6b9c4d66b-l8q2k"] Jul 28 00:31:05.731317 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.731288 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx"] Jul 28 00:31:05.734716 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:05.734696 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e175a-67858f87d7-8pjbx"] Jul 28 00:31:06.214327 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.214297 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:06.404451 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.404381 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms" Jul 28 00:31:06.423700 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.423676 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms"] Jul 28 00:31:06.427522 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.427488 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6e175a-c6df89d68-lsxms"] Jul 28 00:31:06.564879 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.564844 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a18cc2b-4f56-48d2-8cc5-dace579cfdad" path="/var/lib/kubelet/pods/0a18cc2b-4f56-48d2-8cc5-dace579cfdad/volumes" Jul 28 00:31:06.565143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.565131 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a1137c-1853-4d80-a4d8-e4b63361fbda" path="/var/lib/kubelet/pods/66a1137c-1853-4d80-a4d8-e4b63361fbda/volumes" Jul 28 00:31:06.565434 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.565423 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="842e7ea4-32f1-4418-adf3-498ac4383318" path="/var/lib/kubelet/pods/842e7ea4-32f1-4418-adf3-498ac4383318/volumes" Jul 28 00:31:06.565755 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:06.565743 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" path="/var/lib/kubelet/pods/91bf45cf-c03b-4e10-8f45-c46b4d25cdbe/volumes" Jul 28 00:31:09.683935 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:09.683896 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt"] Jul 28 00:31:09.684414 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:09.684243 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:09.685970 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:09.685942 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-6e175a-t2qwt"] Jul 28 00:31:10.095343 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:10.095312 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:10.564711 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:10.564680 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52765958-1ba9-4fd7-9206-b474dbe5e266" path="/var/lib/kubelet/pods/52765958-1ba9-4fd7-9206-b474dbe5e266/volumes" Jul 28 00:31:13.286356 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:13.286319 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:13.349365 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:13.349332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:15.906731 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:15.906698 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:15.970092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:15.970047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:16.192695 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:16.192619 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:16.473070 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:16.472993 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:22.542383 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:22.542350 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4tz9k_ea0be1c2-198c-477c-9762-7a9ddd914902/manager/0.log" Jul 28 00:31:22.666535 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:22.666489 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ld8xt_99ce8fe3-a611-4b1d-8dc6-c69c35823f2b/manager/0.log" Jul 28 00:31:22.788793 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:22.788762 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8b897757-q8hm7_ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a/manager/0.log" Jul 28 00:31:22.915260 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:22.915190 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6dfjv_2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec/server/0.log" Jul 28 00:31:23.036712 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:23.036682 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pnshl_e8b6038f-3e24-4d26-87cd-215ee044673c/manager/0.log" Jul 28 00:31:23.428707 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:23.428680 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79b58bb86f-s8hcv_c53a53de-9d1d-4e58-b38b-e930aa3833fb/copy-manifests/0.log" Jul 28 00:31:23.435720 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:23.435697 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79b58bb86f-s8hcv_c53a53de-9d1d-4e58-b38b-e930aa3833fb/manager/0.log" Jul 28 00:31:23.792699 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:23.792670 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-655bcd7745-gdkb2_d339da45-e0f5-4608-ba77-966d6d175b6b/maas-api/0.log" Jul 28 00:31:24.044280 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:24.044207 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753295-t8htx_07aa5c98-acbc-4786-a8bf-abcac684a963/cleanup/0.log" Jul 28 00:31:24.159349 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:24.159325 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753310-nhjm5_298f7354-fe28-4aeb-8a8d-21c389e24a78/cleanup/0.log" Jul 28 00:31:24.273440 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:24.273414 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xgz87_712c1751-e6c7-4498-b676-15b59ed93e5a/postgres/0.log" Jul 28 00:31:25.745196 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:25.745164 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7dx68_099eb9bd-acdf-4b70-ac5f-9d6010479968/manager/0.log" Jul 28 00:31:25.859178 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:25.859146 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zgm24_e071e05a-5504-4633-ace8-135a5010c156/manager/0.log" Jul 28 00:31:26.091389 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:26.091305 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v7dmc_e62024de-08cd-4702-b8e4-015f95b0a854/registry-server/0.log" Jul 28 00:31:26.473228 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:26.473150 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-blxc4_0a7ffbf8-2622-4788-a433-e3b1caa9559d/manager/0.log" Jul 28 00:31:27.230438 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:27.230407 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f44c56477-pjfvw_c8bc1753-f028-4a6a-97d1-021a0aad7b74/kube-auth-proxy/0.log" Jul 28 00:31:27.478193 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:27.478166 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-p7l7g_0ba737cb-3def-43c3-92f1-cb217791ff3f/payload-pre-processing/0.log" Jul 28 00:31:27.622861 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:27.622834 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tlfb6_cb094531-c0c3-4729-a052-0b2cd6e835d2/payload-processing/0.log" Jul 28 00:31:28.320732 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:28.320696 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6hfjl_4565b4a3-734a-4376-934f-54ad84f5e32e/main/0.log" Jul 28 00:31:28.435031 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:28.435001 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnm9lw_ec627b61-b918-47d3-9f97-49a8ce08d6d7/main/0.log" Jul 28 00:31:28.552601 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:28.552575 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-l42pn_69aaa479-b1a6-40f9-bf5c-9eaac60c7816/main/0.log" Jul 28 00:31:34.818564 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.818526 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjdrz/must-gather-mr94x"] Jul 28 00:31:34.819143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819059 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52765958-1ba9-4fd7-9206-b474dbe5e266" containerName="revoke-keys" Jul 28 00:31:34.819143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819081 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="52765958-1ba9-4fd7-9206-b474dbe5e266" containerName="revoke-keys" Jul 28 00:31:34.819143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819112 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" containerName="payload-processing" Jul 28 00:31:34.819143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819121 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" containerName="payload-processing" Jul 28 00:31:34.819143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819134 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a18cc2b-4f56-48d2-8cc5-dace579cfdad" containerName="payload-pre-processing" Jul 28 00:31:34.819143 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819143 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a18cc2b-4f56-48d2-8cc5-dace579cfdad" containerName="payload-pre-processing" Jul 28 00:31:34.819466 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819157 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842e7ea4-32f1-4418-adf3-498ac4383318" containerName="maas-api" Jul 28 00:31:34.819466 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819166 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="842e7ea4-32f1-4418-adf3-498ac4383318" containerName="maas-api" Jul 28 00:31:34.819466 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819254 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="842e7ea4-32f1-4418-adf3-498ac4383318" containerName="maas-api" Jul 28 00:31:34.819466 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819264 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="52765958-1ba9-4fd7-9206-b474dbe5e266" containerName="revoke-keys" Jul 28 00:31:34.819466 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819276 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a18cc2b-4f56-48d2-8cc5-dace579cfdad" containerName="payload-pre-processing" Jul 28 00:31:34.819466 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.819287 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91bf45cf-c03b-4e10-8f45-c46b4d25cdbe" containerName="payload-processing" Jul 28 00:31:34.822675 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.822653 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:34.824947 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.824925 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjdrz\"/\"openshift-service-ca.crt\"" Jul 28 00:31:34.825079 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.824950 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjdrz\"/\"kube-root-ca.crt\"" Jul 28 00:31:34.825174 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.825160 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mjdrz\"/\"default-dockercfg-m9kzq\"" Jul 28 00:31:34.830428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.830405 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjdrz/must-gather-mr94x"] Jul 28 00:31:34.833390 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.833364 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:34.960632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.960604 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/46ad8362-2057-4e58-8cde-3953a6bf6dd0-must-gather-output\") pod \"must-gather-mr94x\" (UID: \"46ad8362-2057-4e58-8cde-3953a6bf6dd0\") " pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:34.960632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:34.960635 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5k6\" (UniqueName: \"kubernetes.io/projected/46ad8362-2057-4e58-8cde-3953a6bf6dd0-kube-api-access-9z5k6\") pod \"must-gather-mr94x\" (UID: \"46ad8362-2057-4e58-8cde-3953a6bf6dd0\") " pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:35.061969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.061930 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/46ad8362-2057-4e58-8cde-3953a6bf6dd0-must-gather-output\") pod \"must-gather-mr94x\" (UID: \"46ad8362-2057-4e58-8cde-3953a6bf6dd0\") " pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:35.061969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.061971 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5k6\" (UniqueName: \"kubernetes.io/projected/46ad8362-2057-4e58-8cde-3953a6bf6dd0-kube-api-access-9z5k6\") pod \"must-gather-mr94x\" (UID: \"46ad8362-2057-4e58-8cde-3953a6bf6dd0\") " pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:35.062312 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.062291 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/46ad8362-2057-4e58-8cde-3953a6bf6dd0-must-gather-output\") pod \"must-gather-mr94x\" (UID: \"46ad8362-2057-4e58-8cde-3953a6bf6dd0\") " pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:35.069628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.069566 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5k6\" (UniqueName: \"kubernetes.io/projected/46ad8362-2057-4e58-8cde-3953a6bf6dd0-kube-api-access-9z5k6\") pod \"must-gather-mr94x\" (UID: \"46ad8362-2057-4e58-8cde-3953a6bf6dd0\") " pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:35.133188 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.133160 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjdrz/must-gather-mr94x" Jul 28 00:31:35.261245 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.261219 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjdrz/must-gather-mr94x"] Jul 28 00:31:35.262551 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:31:35.262521 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ad8362_2057_4e58_8cde_3953a6bf6dd0.slice/crio-494fc41bca5d60fbf90a3a74dbf03ef083b4646a1bd47b94a3cf356ce799e5eb WatchSource:0}: Error finding container 494fc41bca5d60fbf90a3a74dbf03ef083b4646a1bd47b94a3cf356ce799e5eb: Status 404 returned error can't find the container with id 494fc41bca5d60fbf90a3a74dbf03ef083b4646a1bd47b94a3cf356ce799e5eb Jul 28 00:31:35.264368 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.264353 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:31:35.546322 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:35.546281 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjdrz/must-gather-mr94x" event={"ID":"46ad8362-2057-4e58-8cde-3953a6bf6dd0","Type":"ContainerStarted","Data":"494fc41bca5d60fbf90a3a74dbf03ef083b4646a1bd47b94a3cf356ce799e5eb"} Jul 28 00:31:36.557066 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:36.557025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjdrz/must-gather-mr94x" event={"ID":"46ad8362-2057-4e58-8cde-3953a6bf6dd0","Type":"ContainerStarted","Data":"531278522be9b1203f0ff9829b5ae3693722d0bc2b0d86376cf89b9b04747847"} Jul 28 00:31:36.557451 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:36.557076 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjdrz/must-gather-mr94x" event={"ID":"46ad8362-2057-4e58-8cde-3953a6bf6dd0","Type":"ContainerStarted","Data":"b7493bd50cf367f713e38d10e84b28b25cf95d17f3a8703dedac50df0f586bc3"} Jul 28 00:31:36.573783 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:36.573729 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjdrz/must-gather-mr94x" podStartSLOduration=1.760010216 podStartE2EDuration="2.573711786s" totalImagesPullingTime="813.70157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:31:36.570435366 +0000 UTC m=+3296.494644259" watchObservedRunningTime="2026-07-28 00:31:36.573711786 +0000 UTC m=+3296.497920663" Jul 28 00:31:37.392706 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:37.392673 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:37.602768 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:37.602739 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gqttn_5ab68dc8-3b95-4c0e-9a6c-611abfc63751/global-pull-secret-syncer/0.log" Jul 28 00:31:37.698969 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:37.698857 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" Jul 28 00:31:37.819221 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:37.819178 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-94.ec2.internal_a6d5157fea5c68626415a727705ed8a3/haproxy/0.log" Jul 28 00:31:41.313998 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:41.313960 2555 scope.go:117] "RemoveContainer" containerID="4b622634ebb8ef959b407ee21a805e2f6305f737bcf1594ea6970a079744b876" Jul 28 00:31:42.940088 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:42.940004 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7dx68_099eb9bd-acdf-4b70-ac5f-9d6010479968/manager/0.log" Jul 28 00:31:42.966471 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:42.966432 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zgm24_e071e05a-5504-4633-ace8-135a5010c156/manager/0.log" Jul 28 00:31:43.077405 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:43.077358 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v7dmc_e62024de-08cd-4702-b8e4-015f95b0a854/registry-server/0.log" Jul 28 00:31:43.455025 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:43.454985 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-blxc4_0a7ffbf8-2622-4788-a433-e3b1caa9559d/manager/0.log" Jul 28 00:31:45.570084 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:45.570046 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcxdc_0465816e-82d7-4e52-8780-9e38483682d7/node-exporter/0.log" Jul 28 00:31:45.591112 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:45.591086 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcxdc_0465816e-82d7-4e52-8780-9e38483682d7/kube-rbac-proxy/0.log" Jul 28 00:31:45.616306 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:45.616278 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcxdc_0465816e-82d7-4e52-8780-9e38483682d7/init-textfile/0.log" Jul 28 00:31:47.260241 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.260208 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cmgpx_2d50a704-4e89-4af2-ad0f-d7ad884eaa1e/konnectivity-agent/0.log" limit=1024 Jul 28 00:31:47.746281 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.746247 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff"] Jul 28 00:31:47.754487 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.754460 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.759049 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.758990 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff"] Jul 28 00:31:47.800859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.800470 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-proc\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.800859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.800582 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-sys\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.800859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.800614 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-podres\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.800859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.800672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-lib-modules\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.800859 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.800705 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4797\" (UniqueName: \"kubernetes.io/projected/7200f481-7193-479a-8567-bb23ffe24dce-kube-api-access-f4797\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.902072 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.902029 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-proc\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.902308 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.902081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-sys\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.902936 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.902636 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-podres\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.903049 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.903007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-lib-modules\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.903391 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.903365 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-sys\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.903507 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.903480 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-proc\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.904707 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.904675 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-podres\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.913523 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.904893 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7200f481-7193-479a-8567-bb23ffe24dce-lib-modules\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.913523 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.912923 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4797\" (UniqueName: \"kubernetes.io/projected/7200f481-7193-479a-8567-bb23ffe24dce-kube-api-access-f4797\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:47.921628 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:47.921603 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4797\" (UniqueName: \"kubernetes.io/projected/7200f481-7193-479a-8567-bb23ffe24dce-kube-api-access-f4797\") pod \"perf-node-gather-daemonset-bb6ff\" (UID: \"7200f481-7193-479a-8567-bb23ffe24dce\") " pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:48.070011 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:48.069924 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:48.243862 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:48.243821 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff"] Jul 28 00:31:48.250278 ip-10-0-130-94 kubenswrapper[2555]: W0728 00:31:48.250229 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7200f481_7193_479a_8567_bb23ffe24dce.slice/crio-f09b8c9af54d5e51db3d999bb64945423a54f1823c91b5b974a0ca1c212bb38a WatchSource:0}: Error finding container f09b8c9af54d5e51db3d999bb64945423a54f1823c91b5b974a0ca1c212bb38a: Status 404 returned error can't find the container with id f09b8c9af54d5e51db3d999bb64945423a54f1823c91b5b974a0ca1c212bb38a Jul 28 00:31:48.654442 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:48.654358 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" event={"ID":"7200f481-7193-479a-8567-bb23ffe24dce","Type":"ContainerStarted","Data":"a45096495d3b9e46d67965f4fc9aa70dea3f5db4305d353d590a89ce44921ee7"} Jul 28 00:31:48.654442 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:48.654412 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" event={"ID":"7200f481-7193-479a-8567-bb23ffe24dce","Type":"ContainerStarted","Data":"f09b8c9af54d5e51db3d999bb64945423a54f1823c91b5b974a0ca1c212bb38a"} Jul 28 00:31:48.655003 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:48.654522 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:48.671113 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:48.671058 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" podStartSLOduration=1.671045224 podStartE2EDuration="1.671045224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:31:48.668664363 +0000 UTC m=+3308.592873255" watchObservedRunningTime="2026-07-28 00:31:48.671045224 +0000 UTC m=+3308.595254101" Jul 28 00:31:50.085951 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:50.085927 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqr4s_65b59dc8-4046-499e-97e0-3192fa260310/dns/0.log" Jul 28 00:31:50.110149 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:50.110126 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqr4s_65b59dc8-4046-499e-97e0-3192fa260310/kube-rbac-proxy/0.log" Jul 28 00:31:50.222966 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:50.222928 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n688k_25d14687-0f5b-47e0-9d9c-20c33d00e49e/dns-node-resolver/0.log" Jul 28 00:31:50.774619 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:50.774589 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29753280-bgs47_60da7601-7076-476c-a357-1cc8f6a28fe4/image-pruner/0.log" Jul 28 00:31:50.819848 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:50.819817 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bc54fbf6b-4qdsv_0f6a536d-530c-481f-8db5-3718f487a32f/registry/0.log" Jul 28 00:31:50.886015 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:50.885984 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s7jtw_cc3eb81d-5605-4d0a-a7ec-6e952d769f35/node-ca/0.log" Jul 28 00:31:51.983972 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:51.983942 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f44c56477-pjfvw_c8bc1753-f028-4a6a-97d1-021a0aad7b74/kube-auth-proxy/0.log" Jul 28 00:31:52.071802 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:52.071774 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-p7l7g_0ba737cb-3def-43c3-92f1-cb217791ff3f/payload-pre-processing/0.log" Jul 28 00:31:52.102040 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:52.102012 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tlfb6_cb094531-c0c3-4729-a052-0b2cd6e835d2/payload-processing/0.log" Jul 28 00:31:52.981992 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:52.981960 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2fvf_a89c6a8f-50c8-46bb-ab07-ac6663f43fd9/serve-healthcheck-canary/0.log" Jul 28 00:31:53.462632 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:53.462583 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46fmv_524dce36-43f9-45c7-bf24-6ad7918aeb94/kube-rbac-proxy/0.log" Jul 28 00:31:53.482299 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:53.482272 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46fmv_524dce36-43f9-45c7-bf24-6ad7918aeb94/exporter/0.log" Jul 28 00:31:53.503484 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:53.503452 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46fmv_524dce36-43f9-45c7-bf24-6ad7918aeb94/extractor/0.log" Jul 28 00:31:54.674170 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:54.674140 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mjdrz/perf-node-gather-daemonset-bb6ff" Jul 28 00:31:55.632063 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:55.632035 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4tz9k_ea0be1c2-198c-477c-9762-7a9ddd914902/manager/0.log" Jul 28 00:31:55.922162 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:55.922079 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ld8xt_99ce8fe3-a611-4b1d-8dc6-c69c35823f2b/manager/0.log" Jul 28 00:31:56.146180 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:56.146139 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8b897757-q8hm7_ce7d0a8c-7c2a-4fbc-bd1c-0da3ae14ee4a/manager/0.log" Jul 28 00:31:56.181647 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:56.181533 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6dfjv_2d8c7a77-95b2-4fd1-b93d-1c884eb9fbec/server/0.log" Jul 28 00:31:56.284524 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:56.284069 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pnshl_e8b6038f-3e24-4d26-87cd-215ee044673c/manager/0.log" Jul 28 00:31:56.550992 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:56.550964 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79b58bb86f-s8hcv_c53a53de-9d1d-4e58-b38b-e930aa3833fb/manager/0.log" Jul 28 00:31:56.592833 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:56.592806 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79b58bb86f-s8hcv_c53a53de-9d1d-4e58-b38b-e930aa3833fb/copy-manifests/0.log" Jul 28 00:31:58.246647 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:58.246622 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f44c56477-pjfvw_c8bc1753-f028-4a6a-97d1-021a0aad7b74/kube-auth-proxy/0.log" Jul 28 00:31:58.372867 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:58.372795 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-p7l7g_0ba737cb-3def-43c3-92f1-cb217791ff3f/payload-pre-processing/0.log" Jul 28 00:31:58.411786 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:31:58.411760 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tlfb6_cb094531-c0c3-4729-a052-0b2cd6e835d2/payload-processing/0.log" Jul 28 00:32:00.136033 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:00.136004 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f44c56477-pjfvw_c8bc1753-f028-4a6a-97d1-021a0aad7b74/kube-auth-proxy/0.log" Jul 28 00:32:00.206682 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:00.206651 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-p7l7g_0ba737cb-3def-43c3-92f1-cb217791ff3f/payload-pre-processing/0.log" Jul 28 00:32:00.229397 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:00.229366 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tlfb6_cb094531-c0c3-4729-a052-0b2cd6e835d2/payload-processing/0.log" Jul 28 00:32:03.182428 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:03.182342 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c79pg_80811de6-0dbd-4dad-a545-7b4efd4097c0/migrator/0.log" Jul 28 00:32:03.201956 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:03.201924 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c79pg_80811de6-0dbd-4dad-a545-7b4efd4097c0/graceful-termination/0.log" Jul 28 00:32:04.759131 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.759104 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/kube-multus-additional-cni-plugins/0.log" Jul 28 00:32:04.778927 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.778904 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/egress-router-binary-copy/0.log" Jul 28 00:32:04.799034 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.799003 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/cni-plugins/0.log" Jul 28 00:32:04.820283 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.820254 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/bond-cni-plugin/0.log" Jul 28 00:32:04.840092 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.840064 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/routeoverride-cni/0.log" Jul 28 00:32:04.860717 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.860687 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/whereabouts-cni-bincopy/0.log" Jul 28 00:32:04.883611 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.883583 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pm6m5_3a3a0fc3-cab7-48ac-9e8d-e04de7418400/whereabouts-cni/0.log" Jul 28 00:32:04.934549 ip-10-0-130-94 kubenswrapper[2555]: I0728 00:32:04.934513 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k9p72_45760ac0-5980-4697-aaa1-c7945a19b317/kube-multus/0.log"